pandas interviews should reveal whether a candidate can predict how labels, data types and missing values affect a transformation. Code that looks concise can silently align the wrong rows, expand a join or change the meaning of a missing value. Recruiters should ask candidates to explain both the resulting values and the index, shape and dtypes that accompany them.
These 100+ pandas Interview Questions and Answers cover 120 questions across beginner, intermediate and experienced levels. They move from Series and DataFrame operations to grouped calculations, joins, reshaping, time series, storage and performance. The focus is practical command of the library: choosing an operation, predicting its behavior and checking the result against the intended data contract.
How can recruiters distinguish real pandas fluency from a familiar-looking chain of methods?
Give the candidate a small DataFrame, ask them to predict the output, then introduce duplicate labels, a missing key or a different dtype. A strong explanation follows the row identity and explains which assumptions the operation needs. Asking for a compact assertion or a counterexample also reveals whether the candidate understands the transformation beyond its happy path.
Sherlock AI helps hiring teams assess signals of potential AI cheating, deepfakes, impersonation and unauthorized assistance during remote interviews. Combine those signals with DataFrame walkthroughs and follow-up questions about alignment, types and edge cases. Treat them as context for a fair review alongside demonstrated understanding, rather than as conclusive proof of misconduct.
pandas Interview Questions
Select questions that match the role’s data-processing responsibilities. Provide a small input with explicit labels and types, and ask candidates to predict the result before running code. The complete bank appears first, followed by identically numbered answers.
pandas Interview Questions for Beginners
Check whether the candidate understands DataFrame structure, label-based selection, dtypes and missing-value operations. Ask for the shape and type of a result as well as its visible values.
How do a Series and a DataFrame differ in pandas?
What should you check when constructing a DataFrame from lists?
How do you inspect the structure of an unfamiliar DataFrame?
Why is an integer index label not the same as a row position?
How does loc slicing differ from ordinary Python slicing?
How does iloc handle positional slices and out-of-range access?
When would you use at or iat?
Why do df["score"] and df[["score"]] return different objects?
How should multiple conditions be combined in a pandas boolean mask?
What does DataFrame.query offer, and what are its limits?
How do you make sort_values tie behavior explicit?
How does sort_index differ from sort_values?
What does reset_index do with the old index?
How should set_index be combined with a key-uniqueness check?
How do you rename columns without changing their values?
How do you drop rows or columns by label?
How do head, tail and sample support inspection?
What changed about inferred string dtype in pandas 3.x?
When is astype the right conversion tool?
How does to_numeric help identify invalid numeric text?
Why use the nullable Int64 dtype for integer data with missing values?
How should missing values be detected across pandas dtypes?
Why does using pd.NA directly as a boolean raise an error?
How do subset and thresh change dropna behavior?
How can fillna apply different defaults to different columns?
How do ffill, bfill and their limit parameter work?
How does replace differ from fillna?
How do duplicated and drop_duplicates use subset and keep?
How do normalize and dropna affect value_counts?
How do unique and nunique differ?
How do vectorized string methods handle nullable text?
Why does str.contains need an explicit regex choice for literal searches?
How can str.extract create structured columns from a pattern?
Why specify a format when parsing dates with to_datetime?
What does the dt accessor provide for datetime-like values?
How does Timedelta represent elapsed durations?
Why does to_csv often need index=False?
How can read_csv control columns, identifier types and missing-value parsing?
How does to_dict orient affect the exported structure?
How does equals differ from elementwise equality?
pandas Interview Questions for Intermediate
Explore how the candidate combines transformations without losing row identity. Joins, grouped operations and reshaping are especially useful for testing whether a candidate can anticipate cardinality and alignment.
Why can adding two Series produce missing values even when both inputs contain only numbers?
How does assigning a Series differ from assigning its array to a DataFrame column?
How do you distinguish missing labels introduced by reindex from existing missing values?
When should you align two objects explicitly before a calculation?
How do where and mask differ when replacing selected values?
Can one assign expression use a column created earlier in the same call?
How should chained assignment be rewritten under pandas 3.x Copy-on-Write?
How do shallow and deep DataFrame copies behave under Copy-on-Write?
How does named aggregation produce a stable summary schema?
When is groupby transform preferable to merging an aggregate back?
How does groupby filter differ from filtering individual rows?
How do dropna and observed control categorical groupby output in pandas 3.x?
Does groupby sorting change the order of observations inside each group?
Why do size, count and sum with min_count answer different questions?
How can merge validate prevent silent row multiplication?
How does a pandas merge treat missing keys differently from a typical SQL equality join?
How can merge indicator identify records missing from a lookup?
Why should merge keys and suffixes be explicit when both frames share column names?
Why can concat along columns create extra rows?
What do verify_integrity and ignore_index each guarantee during concatenation?
When should duplicate pivot keys cause an error instead of being aggregated?
How do you melt a wide table while preserving observation identity?
How do stack and unstack move levels without changing the underlying meaning?
How does crosstab normalization change the question a table answers?
What happens when explode encounters empty lists or several list columns?
How do cut and qcut produce different bins from the same values?
How should you change categorical levels without corrupting their meaning?
How do min_periods and closed affect a time-based rolling calculation?
When should an expanding calculation replace a rolling calculation?
How do resample labels differ from resample interval boundaries?
Why is asfreq not a substitute for resample aggregation?
How do tz_localize and tz_convert differ around daylight-saving transitions?
When does a Period represent a value more accurately than a Timestamp?
How does shift with a frequency differ from shifting values by rows?
How does merge_asof choose a nearby timestamp without joining to a future observation?
How do combine_first and update differ when reconciling two frames?
When should a row-wise apply be replaced by a vectorized expression?
What happens when Series.map uses a dictionary that lacks some keys?
How does pipe make a transformation chain easier to test?
Why must frames be aligned before DataFrame.compare can explain their differences?
pandas Interview Questions for Experienced
Probe the candidate’s understanding of memory, representation, version changes and execution boundaries. Ask which assumptions survive a conversion, partition or storage round trip and how they would test that contract.
Why can a NumPy array exported from pandas be read-only?
How would you isolate mutable cell payloads while defining cross-row aliasing?
Why can reindexing fail when duplicate labels are present?
How does MultiIndex sorting affect range slicing?
When would you disallow duplicate labels explicitly?
How can mismatched categorical keys affect a merge?
How do the str and string dtypes differ in pandas 3.x?
What should be checked before adopting Arrow-backed pandas dtypes?
How does nullable Boolean logic affect filtering?
How can numeric dtype choices silently damage exact identifiers or arithmetic?
Why should floating-point reduction tests use meaningful tolerances?
What migration issue does include_groups address in GroupBy.apply?
How can grouped rolling results be safely assigned back to source rows?
When is a custom BaseIndexer appropriate for rolling calculations?
When is the Numba engine a reasonable choice for rolling apply?
How can merge_ordered forward filling change the meaning of a result?
How would you estimate a many-to-many merge before allocating its result?
Why is repeatedly concatenating growing frames inefficient?
Which aggregations can be computed correctly across CSV chunks?
How do you detect CSV schema drift without silently losing valid values?
What must a Parquet round-trip test verify beyond matching values?
How do SQL read and write boundaries affect pandas types and transactions?
How does JSON table orientation differ from records for round trips?
Why is an Excel round trip not a reliable formula or type-preservation test?
What does memory_usage(deep=True) omit from a memory investigation?
When does a sparse dtype save memory, and when can it cost more?
Are pandas string methods always faster than a Python loop?
How can iterrows change the type or precision of a row value?
What are the performance and security limits of pandas eval?
Which operations become incorrect when a frame is processed in independent partitions?
How would you decide whether to move a workload beyond in-memory pandas?
Why should hash_pandas_object not be used as a permanent business identifier?
How should assert_frame_equal express a pipeline’s actual contract?
How does NumPy ufunc behavior change when Series are converted to arrays?
What is lost when a timezone-aware timestamp is serialized as a local clock string?
How should idxmax and idxmin handle ties and entirely missing data?
Why is a business-day offset incomplete without a calendar policy?
How do closed endpoints and overlapping intervals affect IntervalIndex lookups?
What does infer_freq tell you about an irregular or timezone-aware series?
What should a pandas upgrade compatibility matrix include?
pandas Interview Questions and Answers
Examples target pandas 3.1 with NumPy 2.5 unless an answer states another version boundary. pandas 3.x uses Copy-on-Write and changed the default inference of string data, so older examples may behave differently. The answers make relevant dtype, missing-value and alignment choices explicit; optional I/O and execution engines also have their own dependencies.
pandas Interview Questions and Answers for Beginners
These answers build library fluency through small, concrete examples. They explain selection, conversion, string operations and basic I/O while making missing values and index behavior visible.
1. How do a Series and a DataFrame differ in pandas?
A Series is a one-dimensional labeled array, while a DataFrame is a two-dimensional labeled table whose columns can have different dtypes. Both have an index, but a DataFrame also has a column axis. The distinction affects selection results and the methods that are appropriate for them.
Here the same values are represented as a Series and a one-column DataFrame. A one-column DataFrame is still two-dimensional; it is not automatically interchangeable with a Series. I would inspect the expected shape at function boundaries instead of assuming that anything displaying one column has the same type.
2. What should you check when constructing a DataFrame from lists?
Columns supplied as ordinary lists must have compatible lengths, and an explicitly supplied index must match the row count. The constructor cannot infer which missing position was intended when one list is shorter. Constructing from labeled objects follows additional rules, so it should not be confused with positional list construction.
The valid frame has two rows; the second construction raises ValueError. I would validate the upstream records rather than pad a short list with arbitrary values just to satisfy the constructor. A missing measurement and an omitted record are different problems, and the data contract should decide how either is represented.
3. How do you inspect the structure of an unfamiliar DataFrame?
I begin with shape, column labels, dtypes and a small sample, then use info when I need a compact structural summary. These checks reveal whether the object has the expected fields and types before a calculation relies on them.
The identifier remains text and units uses a nullable integer dtype. I would not infer the full schema from head alone: the first rows can hide missing values or malformed data later in the frame. Likewise, a dtype is a storage-level fact, not proof that a column contains semantically valid identifiers or quantities.
4. Why is an integer index label not the same as a row position?
An index label identifies a row on the index axis; a position describes its zero-based location in the current object. Integer labels can make this distinction easy to miss, so explicit loc and iloc access is clearer than relying on ambiguous-looking bracket syntax.
The first row’s label is 10, not 0. Reordering rows changes positions while labels can remain attached to their records. I would choose the accessor that matches the requirement: “customer label 10” is a label lookup, whereas “the first displayed row” is a positional lookup.
5. How does loc slicing differ from ordinary Python slicing?
Loc selects by labels. For a straightforward label slice on a suitable ordered index, both endpoints are included when present, unlike Python’s usual exclusive stop position. Missing, nonmonotonic or duplicate labels can complicate slicing, so the index should be understood before applying a range.
This includes c. I would not translate an iloc slice into loc by changing the accessor alone, because the endpoints have different meanings. When the request is a known list of labels rather than a label interval, passing that explicit list makes the intended selection easier to review.
6. How does iloc handle positional slices and out-of-range access?
Iloc uses zero-based integer positions and follows Python-style slicing with an exclusive stop. A scalar position outside the valid range raises IndexError, while a slice can extend beyond the end and simply return the available positions.
The labels 100, 200 and 300 do not alter these positional rules. A broad slice returning fewer rows than requested can be legitimate, so I would assert the expected length when a downstream calculation requires an exact number of observations.
7. When would you use at or iat?
At and iat are accessors intended for a single scalar value: at uses labels and iat uses integer positions. They express a narrower operation than loc or iloc, which also support slices, masks and lists.
With unique labels, these lookups clearly identify one cell. I would not use a scalar accessor to conceal duplicate index or column labels; if a supposed key is non-unique, that structural problem should be addressed. For selecting several cells, loc or iloc is the appropriate interface rather than a loop of scalar lookups by default.
8. Why do df["score"] and df[["score"]] return different objects?
A single column label normally selects a Series, while a list of column labels selects a DataFrame, even when the list contains only one label. This is a useful way to preserve a two-dimensional shape for an API that expects a table.
This example assumes unique column labels. Duplicate column names can make a single-label selection return more than expected, so I would validate that assumption. The extra brackets are not cosmetic: they pass a list as the selection key and preserve the column axis.
9. How should multiple conditions be combined in a pandas boolean mask?
Use elementwise operators such as & and |, with each comparison parenthesized. Python’s and and or try to evaluate the truth value of the entire object, which is ambiguous for a Series rather than a row-by-row operation.
Only the third row meets both conditions. Parentheses also prevent operator-precedence mistakes in expressions such as comparisons combined with &. I would inspect missing-value behavior when the inputs are nullable, and decide explicitly whether an unknown condition should be excluded or handled separately.
10. What does DataFrame.query offer, and what are its limits?
Query expresses row filtering using a string expression that can refer to column names. It can make a simple filter readable, and an @ prefix can reference a local Python variable. It should not be used to evaluate untrusted expression strings.
This selects the same rows as an explicit boolean mask. Query has its own parser and engine behavior, and names containing spaces require special syntax, so it is not identical to arbitrary Python code. I would prefer a direct mask when it makes the logic clearer or when dynamically constructing a string would obscure the actual condition.
11. How do you make sort_values tie behavior explicit?
Sort_values orders rows by column values. When several rows share a value, either add a meaningful tie-breaker or choose a stable sort when preserving their existing order is the intended rule. A stable order is still based on the input order, which must itself be meaningful.
The stable single-column sort keeps IDs 3 and 1 in their original tied order; the explicit tie-breaker places ID 1 first. I would use a business tie rule for reproducible ranking across differently ordered inputs, rather than assume that stability alone defines a unique result.
12. How does sort_index differ from sort_values?
Sort_index orders the axis labels, while sort_values orders data values in the specified columns or Series. The choice depends on whether the desired order is defined by the labels or by a measured attribute.
The two operations produce different orders because labels and values encode different information. I would not sort the index merely to make a value-based report look ordered. If the index represents timestamps or another meaningful key, index sorting can instead be the exact prerequisite a later label-range operation needs.
13. What does reset_index do with the old index?
Reset_index replaces the current index with a default integer index and, by default, moves the old index into columns. Using drop=True discards the old index values instead. The choice should follow whether those labels contain useful data.
Here the names are meaningful, so retaining them is appropriate if the output needs them. Resetting an index does not sort the rows or remove duplicate records. I would also watch for a naming collision when the old index name already exists as a column.
14. How should set_index be combined with a key-uniqueness check?
The set_index method moves one or more columns into the index, but pandas permits duplicate labels unless the workflow explicitly rejects them. The verify_integrity parameter has been deprecated since pandas 3.0; checking the resulting index’s is_unique property makes the required validation explicit.
The check is useful when later lookups assume one row per ID. It does not validate every business rule or prevent future operations from introducing duplicates. By default the source key column is removed from the columns when it becomes the index; use drop=False only when retaining both representations is intentional.
15. How do you rename columns without changing their values?
Rename can map existing labels to new labels while leaving the underlying data values unchanged. Use the columns argument for column names and the index argument for row labels, making the target axis explicit.
This returns a renamed object while the original still has qty. Errors="raise" helps detect a misspelled source label instead of silently ignoring it. I would also check that the new labels remain unique when uniqueness is expected; renaming two different columns to the same name can make later selection ambiguous.
16. How do you drop rows or columns by label?
Drop removes labels from the specified axis; it does not remove rows by their displayed position unless those positions happen to equal the labels. The index and columns arguments make the intended operation clear.
The row labeled 10 is removed, leaving label 20 rather than automatically renumbering it. A missing label normally raises an error; errors="ignore" can be appropriate for a deliberately optional field, but it can also conceal schema drift. I would not use it simply to silence an unexpected missing-column failure.
17. How do head, tail and sample support inspection?
Head and tail select the first and last rows in the object’s current order. Sample selects a random subset and accepts random_state to make a particular sampling setup reproducible. None of these alone proves the whole dataset is clean or representative.
The fixed seed reproduces the sample for this input and environment; changing input order or library behavior can change the selected records. Head can overrepresent an ordered segment, while a small random sample can miss rare problems. I would combine inspection with targeted validation of important fields.
18. What changed about inferred string dtype in pandas 3.x?
Pandas 3.x infers a dedicated string dtype for ordinary string data instead of defaulting such data to object dtype. This changes code that used dtype==object as a proxy for “text.” The inferred str dtype uses missing-value behavior that differs from explicitly requesting the nullable string dtype in some operations.
I would use pandas’ type-aware checks and explicit schema requirements rather than assume all text columns are object. Storage can depend on available optional dependencies, so tests should focus on the required semantics unless the backend itself is part of the contract. Migration checks should include missing values and operations that previously accepted arbitrary non-string objects.
19. When is astype the right conversion tool?
Astype requests a specific dtype for an object or selected DataFrame columns. It is appropriate when the data is expected to satisfy that target representation; invalid values can raise an error rather than being silently cleaned.
This distinguishes a schema conversion from a data-repair policy. If malformed numeric text is expected, to_numeric with an explicit error policy may be more suitable. I would not cast identifiers to numbers simply because they contain digits, since leading zeros and identity semantics may be lost.
20. How does to_numeric help identify invalid numeric text?
To_numeric parses values as numbers and supports an explicit error policy. Errors="coerce" turns unparseable values into missing values, which can be useful for diagnosis if the original text is retained and invalid input is distinguished from already missing input.
Only bad is identified as an invalid supplied value; None was already missing. I would report or quarantine those invalid values according to the data contract instead of silently treating them as ordinary missing measurements. Successful conversion also does not establish that a number lies within a valid business range.
21. Why use the nullable Int64 dtype for integer data with missing values?
The capital-I Int64 extension dtype can represent integer values and missing entries without converting the entire column to a floating-point dtype. It differs from the NumPy-style int64 dtype, which does not natively represent a missing integer value.
This preserves the distinction between an integer count and a missing count. I would specify the dtype when that distinction is part of the schema rather than depend on constructor inference. Nullable integer storage does not mean all downstream libraries understand pd.NA, so conversion boundaries still need a documented missing-value policy.
22. How should missing values be detected across pandas dtypes?
Use isna or notna rather than equality comparisons with a missing sentinel. Pandas uses different missing representations for different dtypes, and values such as NaN do not obey ordinary equality expectations.
The same API handles the missing integer and missing datetime. Empty strings and sentinel codes such as -999 are not automatically missing unless the ingestion or cleaning policy converts them. I would define those domain-specific representations explicitly instead of assuming isna knows the business meaning of every unusual value.
23. Why does using pd.NA directly as a boolean raise an error?
Pd.NA represents an unknown value, so converting it to one definite True or False would invent information. Pandas therefore raises an error for its direct boolean interpretation. Nullable boolean expressions can retain unknown results until the code applies an explicit policy.
Here filling unknown with False is an explicit example policy, not a universal recommendation. A filter may instead need to report unknown rows separately. I would avoid writing an ordinary Python if around a nullable scalar until the missing case has been handled deliberately.
24. How do subset and thresh change dropna behavior?
Subset limits which columns are examined when dropping rows, while thresh requires a minimum number of non-missing values among the considered fields. These options let the removal rule match the fields actually needed for a particular calculation.
The first selection keeps rows with x present; the second requires both measurements. I would not combine incompatible how and thresh options or drop rows based on unrelated optional columns. The removed population should be measured and reported when filtering can bias the analysis.
25. How can fillna apply different defaults to different columns?
For a DataFrame, a mapping passed to fillna can provide a separate replacement for each named column. This is clearer than replacing every missing value with one scalar when the columns have different meanings and dtypes.
This example assumes the domain permits zero for missing units and unknown for missing labels. If an unrecorded quantity is genuinely unknown, zero would be misleading. I would choose defaults from the data contract, preserve an indicator when imputation needs traceability and confirm the replacement is compatible with the column dtype.
26. How do ffill, bfill and their limit parameter work?
Ffill propagates an earlier available value forward, while bfill propagates a later value backward. Limit caps the number of consecutive missing entries filled in a run. These operations follow the current row order, so ordering and group boundaries must be correct before propagation.
Only one missing entry in each direction is filled. Carrying a measurement forward assumes the previous value remains applicable, and carrying it backward can use information from the future. I would not apply either method across unrelated entities or an unverified sort order merely because it removes missing values.
27. How does replace differ from fillna?
Replace matches specified values or patterns, while fillna targets values pandas already recognizes as missing. Replace is useful for converting a documented sentinel into the missing-value representation before applying a separate missing-data policy.
The sentinel -999 was an ordinary integer until the explicit replacement. The final zero fill is a separate assumption and should only be used if justified. I would avoid broad replacements that can alter valid values or text fragments unintentionally, especially when enabling regular-expression matching.
28. How do duplicated and drop_duplicates use subset and keep?
Duplicated creates a boolean mask identifying repeated rows under the chosen columns, while drop_duplicates removes them according to the same key and retention rule. Subset defines which columns establish equality; keep chooses first, last or no retained duplicate occurrence.
Keep="last" means last in the current row order, not latest by a timestamp unless the data has first been ordered accordingly. I would define the business key and retention rule before removing records. Two similar rows can be legitimate repeated events, so deduplication should not be used as an unexplained cleanup reflex.
29. How do normalize and dropna affect value_counts?
Value_counts summarizes frequencies. Normalize=True returns proportions, and dropna controls whether missing values participate. The denominator changes when missing values are excluded, so the option affects interpretation rather than only display.
A accounts for half of all four observations but two-thirds of the three non-missing observations. I would label the denominator and missing-value treatment in the result. A sorted frequency table is useful for inspection, but category order may need an explicit business ordering for presentation.
30. How do unique and nunique differ?
Unique returns the distinct values in order of appearance rather than sorting them by default. Nunique returns a count of distinct values and normally excludes missing values unless dropna=False is requested. Their outputs answer different questions.
Unique can return an extension array depending on the dtype, so I would not assume its result is always a plain Python list or NumPy array. Nunique is convenient for a cardinality check, but a low number of distinct values does not by itself prove a field is a valid category or key.
31. How do vectorized string methods handle nullable text?
The str accessor applies supported text operations across a Series or Index, with missing-value behavior determined by the method and dtype. It is usually clearer than a Python loop for routine text cleaning, while preserving the distinction between missing and present text.
The missing entry remains missing rather than becoming the literal string “None.” I would avoid converting an entire column with Python str merely to enable text operations, because that can turn missing values into ordinary text. Normalization rules such as lowercasing should also be appropriate to the domain, especially for case-sensitive identifiers.
32. Why does str.contains need an explicit regex choice for literal searches?
Str.contains treats its pattern as a regular expression by default. Characters such as a dot therefore have special meanings unless regex=False is supplied. The na argument lets a filtering policy specify how missing text should be handled.
The literal search finds the actual dot, while the regular expression dot matches a character in both non-missing strings. I would set regex=False for user-supplied literal terms rather than accidentally interpret them as patterns. Choosing na=False excludes missing values; other workflows may need to preserve them for review.
33. How can str.extract create structured columns from a pattern?
Str.extract returns captured groups from a regular-expression match. Named capture groups become useful output column names, and nonmatching values produce missing extracted fields. It extracts the first match rather than every occurrence.
Anchors make this pattern validate the whole string rather than accept a matching fragment inside unrelated text. The extracted number is still text until explicitly converted. I would inspect nonmatches and preserve the source value rather than assume extraction failure means the source record can be discarded.
34. Why specify a format when parsing dates with to_datetime?
An explicit format documents the expected input representation and avoids ambiguity between day-first and month-first interpretations. Errors="coerce" can mark invalid input as NaT, but the original strings should be retained if parsing failures need explanation.
The first value is April 3 under the stated format. A different business format could interpret it as March 4, so inference is not a substitute for knowing the source contract. Time-zone handling is a separate decision; parsing a date successfully does not establish which zone a local timestamp represents.
35. What does the dt accessor provide for datetime-like values?
The dt accessor exposes datetime-like properties and methods on a Series, such as year, month and day-of-week. It requires a suitable datetime-like dtype rather than arbitrary strings that happen to look like dates.
The day_of_week property uses Monday=0, so January 1 here is Thursday and February 2 is Monday. I would document that numbering convention before exporting it to another system. Extracting components can discard context: month alone combines years, and local calendar components depend on the timestamp’s time zone.
36. How does Timedelta represent elapsed durations?
Timedelta represents a duration rather than a calendar label. Subtracting compatible timestamps produces a duration, and total_seconds expresses that full elapsed interval in seconds. The seconds component alone is not the complete duration when days are present.
The interval is 25.5 hours, not 1.5 hours. Calendar concepts such as “next month” have variable lengths and should not be replaced with a fixed 30-day duration without a business justification. Time-zone-aware arithmetic also needs compatible timestamps and an explicit distinction between elapsed time and local calendar time.
37. Why does to_csv often need index=False?
To_csv includes the index by default. If the index is only an incidental row label, exporting it can create an unwanted extra column when the CSV is read elsewhere. Index=False omits that axis from the file while retaining the DataFrame columns.
The index label 99 is intentionally absent, while the identifier’s leading zeros are preserved by the explicit read dtype. If the index contains meaningful keys, keep it or reset it into a named column deliberately. CSV does not preserve a complete pandas schema, so round-trip expectations need explicit types and missing-value conventions.
38. How can read_csv control columns, identifier types and missing-value parsing?
Use usecols to select needed fields, dtype to preserve their intended representation and NA options to define which strings mean missing. These controls prevent inference from silently turning identifiers into numbers or valid text codes into missing values.
Here NA is a legitimate region code, while an empty region is missing. The unused units column is not loaded. I would define these rules from the source contract and verify the selected columns afterward; usecols is a selection feature, not a guarantee that the resulting column order matches the order of the supplied list.
39. How does to_dict orient affect the exported structure?
The orient argument chooses how rows, columns and values are arranged in the resulting Python object. Records produces a list of row dictionaries, while list produces a dictionary mapping columns to lists. The best form depends on the consumer’s contract.
These two orientations do not preserve the row index. If labels must survive, choose an appropriate orientation or materialize them as columns first. To_dict also does not guarantee that every value is JSON-serializable; timestamps and missing sentinels may need an explicit serialization policy at the boundary.
40. How does equals differ from elementwise equality?
Equals returns one boolean indicating whether two pandas objects have the same shape and matching contents under its documented rules, treating missing values in corresponding positions as equal. The == operator instead produces elementwise comparisons and does not directly answer whether the entire objects are equivalent.
The first pair is equal despite the missing values, while the final pair differs in data dtype. For tests requiring detailed diagnostics, pandas.testing.assert_frame_equal is often more informative than a single boolean. I would choose equality rules deliberately, including whether dtype differences or numeric tolerance are acceptable.
pandas Interview Questions and Answers for Intermediate
The intermediate answers work through alignment, assignment, grouped calculations, joins and time-based operations. Expected results and failure cases show why explicit options often matter as much as the method name.
41. Why can adding two Series produce missing values even when both inputs contain only numbers?
Series arithmetic matches index labels, not row positions. The result includes labels present in either input; a label without a counterpart usually produces a missing result. Inspect the index before diagnosing a numeric calculation as incorrect.
Only label b has both operands, so its result is 22. If positions genuinely represent the same observations, positional array arithmetic may be appropriate, but removing labels discards an integrity check. Prefer establishing a stable business index or explicitly validating row order before making that choice; sorting each input independently does not prove the rows correspond.
42. How does assigning a Series differ from assigning its array to a DataFrame column?
A Series assignment aligns its index to the DataFrame index. An array-like assignment uses positional values and must have a compatible length. The same displayed values can therefore produce different columns when the Series index is in a different order.
The aligned column places 9 against east because that is its label. The positional column places 7 in the first row regardless of its original label. Use positional assignment only when row correspondence is an explicit invariant. A length match alone cannot detect a reversed, filtered or independently sorted input.
43. How do you distinguish missing labels introduced by reindex from existing missing values?
reindex conforms an object to requested labels and introduces missing positions for labels not already present. Its fill_value supplies those new positions; it does not generally replace missing values already stored at existing labels.
The distinction preserves provenance: b is an observed label with an unknown value, while c is a newly requested label. Filling both with zero would assert that unknown and absent have the same meaning. Reindexing is also not a duplicate-resolution policy; resolve ambiguous repeated labels before using it to enforce a unique reporting grid.
44. When should you align two objects explicitly before a calculation?
Use align when the intended label set should be visible and reusable before several calculations. Choose an outer, inner, left or right join deliberately, then decide whether replacing missing values is meaningful for the measure.
This treats an absent contribution as zero and retains both labels. An inner alignment would contain no observations here. Unlike a claim that every missing value is zero, the chosen policy needs a domain justification: zero may suit an absent count but not an unmeasured temperature. Retain missingness indicators or compare label sets first when unmatched records need investigation.
45. How do where and mask differ when replacing selected values?
where keeps values where its condition is true and replaces the rest. mask replaces values where its condition is true. Express the condition in terms of what should be retained or replaced, then use the corresponding method to avoid an inverted cleaning rule.
Both expressions clamp negative values to zero in this fixture. Whether that is valid cleaning depends on the field: a negative account adjustment may be legitimate. Check how missing Boolean conditions and label alignment should behave in the actual data, and preserve an audit flag if replacement would otherwise conceal invalid source values.
46. Can one assign expression use a column created earlier in the same call?
Yes. Keyword assignments are evaluated in order, and callable values receive the DataFrame as it exists at that point. Use callables when a later expression depends on a newly created column rather than referencing a name that does not exist in the original frame.
The second callable can read tax because the first assignment has already added it. Keep these functions free of mutations to the input object. Long chains of interdependent expressions can become hard to inspect; name an intermediate result or extract a function when doing so makes units, rounding or business assumptions clearer.
47. How should chained assignment be rewritten under pandas 3.x Copy-on-Write?
Perform the selection and assignment on the intended owner in one operation, usually with loc. Under Copy-on-Write, assigning through a temporary selected object does not reliably mean “update the original,” and chained assignment is not a supported way to express that intent.
This updates only the second row in the original DataFrame. If the intended output is an independently edited subset, create and name that subset instead. Do not suppress a chained-assignment diagnostic as a fix: rewrite ownership explicitly so the code says which object is supposed to change. These examples use pandas 3.1, where Copy-on-Write is the normal model.
48. How do shallow and deep DataFrame copies behave under Copy-on-Write?
In pandas 3.x, a shallow copy can initially share underlying storage, but ordinary pandas writes to either object preserve the other object’s values through Copy-on-Write. A deep copy copies the data more eagerly. Neither option recursively clones arbitrary mutable Python objects stored inside cells.
Choose a copy policy based on ownership and memory needs, not on old assumptions that a shallow copy always propagates edits back to its source. A cell containing a list is a different case: changing that list in place is a Python-object mutation, not a pandas scalar assignment. Avoid mutable objects in cells when independent copies are part of the data contract.
49. How does named aggregation produce a stable summary schema?
Named aggregation specifies each output column together with its input column and reduction. It makes the result’s schema explicit and avoids a MultiIndex of column names when several statistics are calculated together.
The output has one row per team, so it should not be assigned directly back to a three-row input as though it were row-level data. Choose names that state the measure and units rather than exposing the implementation function alone. Specify missing-key and category behavior separately when those cases can occur; a neat schema does not guarantee the intended population was grouped.
50. When is groupby transform preferable to merging an aggregate back?
Use transform for a group-derived value that must line up with every original row. A scalar group result, such as a mean or total, is broadcast to that group’s rows, preserving the original row-level shape.
This avoids an unnecessary join and its possible row multiplication. The denominator still needs validation: a zero group total makes a ratio undefined, and negative adjustments can make a “share” difficult to interpret. Use aggregation when the output itself should have one row per group; use transformation when the group statistic is a feature attached to the original observations.
51. How does groupby filter differ from filtering individual rows?
GroupBy.filter evaluates a rule for each whole group and retains or removes that group’s rows. A row-level Boolean mask evaluates each observation independently. The distinction matters when eligibility depends on a group’s size or total.
Team a keeps both its low and high value because the group has two rows; team b loses its value of 20 because its group has only one. For simple group rules, a built-in transform followed by a mask can be clearer or faster than a Python callback. State whether eligibility is based on all source rows or on a previously filtered subset, because changing that order changes the result.
52. How do dropna and observed control categorical groupby output in pandas 3.x?
dropna controls whether missing grouping keys form a group. observed controls whether unused categorical levels appear. These are separate choices, and pandas 3.x defaults to observed categorical groups; specify both when a report requires stable population rules.
The expanded output includes west even though there was no west observation; its sum is zero under the default sum behavior. That synthetic zero should not be mistaken for a measured zero. Retain a count alongside totals when distinguishing absent groups from groups with observations is important, and define whether unknown region belongs in the report.
53. Does groupby sorting change the order of observations inside each group?
The sort argument primarily controls the order of group keys in the result. Grouping preserves observation order within each group; it does not sort each group by time or another value. Order-dependent operations therefore need an explicit prior sort.
The b observations remain 8 then 1. If the business question is the latest event per team, sort by the event timestamp and a deterministic tie-breaker before taking the relevant row. Turning group-key sorting on does not solve tied timestamps or an unspecified event order.
54. Why do size, count and sum with min_count answer different questions?
size counts rows, count counts nonmissing values in the selected column, and sum(min_count=1) requires at least one valid value before returning a numeric total. Use them together to distinguish absent measurements from genuine zero totals.
Group a has two records but no measured value, while group b has a measured zero. A default sum that returns zero for an all-missing group can erase that difference. Choose the validity threshold from the metric definition; one valid observation may still be insufficient for a complete monthly total.
55. How can merge validate prevent silent row multiplication?
Tell merge the expected key relationship with validate, such as many-to-one for transaction rows joined to a unique lookup. It checks the relevant key uniqueness and raises an error when the contract is violated, rather than silently expanding the result.
Validation does not ensure every transaction found a lookup row; check coverage separately. Avoid “fixing” a failed uniqueness contract by arbitrarily dropping duplicates. Find whether the lookup needs an effective date, a composite key or a documented winner rule, then join at the correct grain.
56. How does a pandas merge treat missing keys differently from a typical SQL equality join?
pandas can match missing join keys to other missing keys. A typical SQL equality join does not treat NULL as equal to NULL. This difference can join unrelated unknown entities and can multiply rows when both sides contain several missing keys.
Here the missing-key left row receives y=30. If unknown keys must never match, separate or filter missing keys before joining and restore the unmatched rows according to the output contract. Do not replace every unknown with a shared sentinel and assume that prevents matching; it creates an ordinary common key with the same conceptual risk.
57. How can merge indicator identify records missing from a lookup?
Use indicator=True to retain the match provenance, then select the appropriate category such as left_only. This distinguishes a missing match from a matched row whose payload happens to contain missing values.
User 3 matched despite its missing name, so filtering on name.isna() would incorrectly classify it as absent. Keep join-key null semantics in mind, and report missing-lookup counts against the intended source population. For a full reconciliation, an outer merge also exposes right-only records.
58. Why should merge keys and suffixes be explicit when both frames share column names?
Explicit keys prevent unrelated same-named columns from silently becoming join criteria. Explicit suffixes preserve the meaning of overlapping payload columns, making conflicts visible for review rather than leaving ambiguous names downstream.
The two identifier columns remain available to inspect because they have different names. If a key lives in an index, request the corresponding index join explicitly rather than assuming indexes always participate in column merges. Decide which status is authoritative with a business rule; renaming overlapping columns does not resolve their disagreement.
59. Why can concat along columns create extra rows?
With axis=1, concatenation aligns row indexes. The default outer join retains the union of row labels, so differently indexed inputs can produce more rows and missing cells. It does not pair the first row of one frame with the first row of another by position.
An inner join would retain only b, which may silently discard needed observations. Resetting both indexes can force positional correspondence, but only do that when the rows are demonstrably paired. If a shared business identifier defines correspondence, use an explicit validated merge instead of relying on incidental indexes.
60. What do verify_integrity and ignore_index each guarantee during concatenation?
verify_integrity checks for overlapping labels on the concatenated axis. ignore_index discards the existing labels on that axis and creates a new default index. Neither option checks duplicate business records stored in ordinary columns.
The second operation has a valid unique generated index but still contains a repeated id. Validate the actual business key separately before treating a concatenated dataset as deduplicated. If the original index identifies source rows, preserve it in a column or with source keys before discarding it.
61. When should duplicate pivot keys cause an error instead of being aggregated?
Use pivot when each index-column pair must identify one value. Use pivot_table when repeated pairs are expected and a defined aggregation is part of the question. Switching methods just to remove an error can hide a violated uniqueness assumption.
The sum is correct only if the two records are separate additive sales contributions. If they are duplicate extracts of the same sale, summing doubles the value. Specify the aggregation explicitly rather than accepting a default mean, and verify which dimensions were removed when defining the new grain.
62. How do you melt a wide table while preserving observation identity?
Place stable identifiers in id_vars and name the columns being converted with value_vars. Melting turns those columns into a variable-value pair, repeating the identifiers for each original measurement.
The resulting grain is one id-month pair. If id was already repeated in the wide input, that pair may not be unique; preserve an additional source-row identifier when needed. Do not melt unrelated columns with incompatible units into one measure merely because their storage format is numeric.
63. How do stack and unstack move levels without changing the underlying meaning?
unstack moves a row-index level into columns; stack moves a column level into the row index. Name the level you intend to move so a change in index-level order does not silently reshape the wrong dimension.
The missing b-Feb cell arises because the rectangular representation includes an unobserved combination. In pandas 3.1, the current stack implementation retains this existing missing cell; do not assume historical dropna behavior without checking the version and API. A round trip can therefore expose additional missing combinations rather than preserve the original sparse index exactly. Decide whether those combinations belong in the dataset.
64. How does crosstab normalization change the question a table answers?
Normalization chooses the denominator: the whole table, each row or each column. State that denominator alongside the result because the same count can represent different conditional proportions.
The web result is the fraction of web observations with a yes outcome, not web’s share of all yes outcomes. Keep counts available because 100% from one store observation carries different information from 100% across thousands. Missing categories and repeated observations can also alter the denominator, so define the eligible unit before calculating a cross-tabulation.
65. What happens when explode encounters empty lists or several list columns?
explode expands list elements into rows while repeating the other fields. An empty list produces a row with a missing exploded value. When exploding several columns together, their element counts must agree within each row so elements can be paired.
Do not count the retained empty-list row as a real item. Validate paired list lengths before expansion, and distinguish pairing from a Cartesian product: separately exploding two columns can create every combination, which is a different operation. Avoid sets when element order matters because their iteration order is not a stable analytical contract.
66. How do cut and qcut produce different bins from the same values?
cut groups by value intervals, while qcut derives quantile boundaries from the observed distribution. Fixed business thresholds and roughly equal-sized population buckets answer different questions.
The value 3 is low under the fixed thresholds but belongs to the upper half of this sample. Quantile boundaries can shift between datasets, and repeated values can prevent distinct quantile edges; do not silently drop duplicate edges and still describe the result as the requested number of buckets. Preserve boundaries when applying a learned binning scheme to later data.
67. How should you change categorical levels without corrupting their meaning?
An ordered categorical uses its declared category order, not lexical order. Add a new level before assigning it, and distinguish removing unused categories from removing a category that still has observations: the latter turns those observations into missing values.
Adding urgent appends it to the category order, which is appropriate only if it should rank after high. Use an explicit reorder when the business order is different. A category migration is a schema change: check values that would become unknown before applying it, rather than interpreting the resulting missingness as source-data absence.
68. How do min_periods and closed affect a time-based rolling calculation?
min_periods requires a minimum number of valid observations before producing a result. closed chooses which interval endpoints belong to a window. A two-day window is a time interval, not necessarily two rows.
At January 3, the right-closed interval excludes January 1 while the both-closed interval includes it. Irregular sampling or duplicate timestamps can change how many rows fall inside the same duration. Sort the time axis, define endpoint semantics and decide whether missing observations mean insufficient coverage before interpreting a rolling total.
69. When should an expanding calculation replace a rolling calculation?
Use expanding windows when every earlier observation remains part of the accumulated history. Use rolling windows when old observations should leave the calculation after a fixed count or duration. The distinction changes the meaning of the metric, not just its smoothness.
The final expanding mean includes all three values; the final rolling mean includes only 4 and 12. Set a meaningful minimum observation count, and reset or group the calculation when a new entity or reporting lifecycle begins. Otherwise an expanding statistic can accidentally carry one customer’s history into another’s.
70. How do resample labels differ from resample interval boundaries?
closed controls which boundary is included in a resampling bin; label chooses the timestamp used to name that bin. Labeling a bin with its right edge does not by itself make the bin right-closed.
The event at 01:00 belongs to the interval beginning at 01:00 and is labeled 02:00. Test events exactly on boundaries when validating reporting bins. For offset or business-calendar bins, also specify the relevant origin, offset or calendar rule instead of assuming midnight alignment fits the reporting period.
71. Why is asfreq not a substitute for resample aggregation?
asfreq conforms a time index to a frequency, introducing missing timestamps without combining observations. resample groups observations into time bins and then applies an aggregation. Choose according to whether the task is creating a grid or summarizing events.
The missing January 2 value is not automatically zero. A sensor gap means something different from a day with no sales in a complete event feed. If several observations share a target interval, resampling can aggregate them; simply selecting grid timestamps may omit off-grid observations or fail on ambiguous duplicate labels.
72. How do tz_localize and tz_convert differ around daylight-saving transitions?
tz_localize interprets naive wall-clock timestamps in a timezone. tz_convert converts already timezone-aware instants to another timezone. Localization needs a policy for nonexistent spring-forward times and ambiguous repeated autumn times.
This deliberately moves a nonexistent 02:30 to the first valid time, 03:00. That is an explicit correction, not recovery of a uniquely known event instant. For ambiguous autumn times, use source offset information or an explicit policy rather than guessing; preserve rejected or corrected records when auditability matters. Converting a naive timestamp cannot invent its original timezone.
73. When does a Period represent a value more accurately than a Timestamp?
A Period represents a span at a frequency, such as a calendar month, while a Timestamp identifies a point in time. Monthly accounting data often needs the period identity rather than an arbitrary midnight chosen to stand for the whole month.
Converting a period to a timestamp requires choosing a boundary; it does not turn the monthly value into a measurement taken at that instant. Be explicit about fiscal frequencies and timezone assumptions when joining period totals to timestamped events. A month is not a fixed number of seconds, so use calendar-aware operations for calendar-based reporting.
74. How does shift with a frequency differ from shifting values by rows?
Without a frequency, shift moves values relative to the existing index and introduces missing positions. With a frequency on a suitable time index, it shifts the index labels while keeping the values associated with their moved timestamps.
A one-row lag is not necessarily a one-day lag when dates are missing. Reindex to the intended calendar or use an explicit timestamp match when the requirement is a previous calendar period. Within grouped data, sort and shift separately for each entity so one entity’s last observation does not become another’s lag.
75. How does merge_asof choose a nearby timestamp without joining to a future observation?
Use direction="backward" to select the latest right-side key at or before the left key, and set a tolerance to reject stale matches. Both sides must be sorted by the ordered merge key. Add entity grouping with by when observations belong to separate streams.
The 10:02 observation finds 10:00; the 10:10 observation rejects 10:05 as five minutes old. Decide whether exact matches are allowed and how duplicate right-side keys should be resolved. A nearest match may select a future observation, so it is inappropriate when the feature must represent information available at the decision time.
76. How do combine_first and update differ when reconciling two frames?
combine_first returns a new object that fills missing values from the other object and can include the union of labels. update modifies the caller using aligned nonmissing values from the other object, without extending the caller’s axes. Its default behavior can overwrite existing nonmissing values.
Use overwrite=False with update when only missing caller values should be filled. Neither method decides which source is authoritative: document whether a newer value should replace an existing one and whether new entities should be admitted. Check conflicts before applying a patch if silent replacement would erase evidence.
77. When should a row-wise apply be replaced by a vectorized expression?
Prefer a direct column expression when it states the same calculation clearly. A row-wise Python callback incurs per-row overhead and can obscure dtype or missing-value behavior. Use apply when a genuinely custom operation cannot be expressed cleanly with suitable built-ins.
This demonstrates equivalent results on the fixture, not a universal performance measurement. Validate nullable values, dtypes and overflow behavior on representative data before substituting one implementation for another. Calling a method “vectorized” does not guarantee it is fast for every dtype; benchmark the actual workload when performance is the reason for changing it.
78. What happens when Series.map uses a dictionary that lacks some keys?
An ordinary dictionary mapping replaces recognized values and yields missing values for unrecognized keys. That can be useful for detecting unmapped codes, but it can also erase the distinction between an unknown source value and an already missing source value.
The unknown flag identifies B without treating the source null as an unmapped code. If unrecognized values should remain unchanged, choose an explicit fallback or a replacement operation with the appropriate semantics. Avoid a blanket fill that labels every missing result “other” unless unknown codes and missing inputs genuinely belong to the same category.
79. How does pipe make a transformation chain easier to test?
pipe passes the current object to a function so a named transformation can participate in a method chain. The benefit is an explicit, independently testable contract, not a change in execution speed or automatic protection against side effects.
The function returns a new result and leaves the input schema unchanged. Test the function’s required columns, units and missing-value policy separately from the surrounding chain. Avoid hiding external reads, mutable global state or in-place edits inside a pipe function, because those make a visually linear pipeline harder to reproduce.
80. Why must frames be aligned before DataFrame.compare can explain their differences?
compare expects identically labeled frames and reports changed values at corresponding positions. It is a value-difference tool, not an entity-matching algorithm. Establish the intended index and column correspondence before asking it to explain changes.
Only b changed once labels are aligned; a’s value remains 1 despite its different source position. Audit added and removed labels separately before reindexing, since reindexing to one side can discard new entities. Matching missing values are not reported as changes, and a compact diff does not replace checks of schema, dtype and key uniqueness.
pandas Interview Questions and Answers for Experienced
Experienced answers examine advanced indexing, extension types, memory behavior, specialized windows and I/O boundaries. They separate correctness guarantees from performance hypotheses and explain where dependency choices or data shape change the answer.
81. Why can a NumPy array exported from pandas be read-only?
Under pandas 3.x Copy-on-Write, to_numpy() can return a read-only view when the result shares underlying storage. This prevents an external array write from silently changing a pandas object. A conversion requiring new storage can behave differently, so neither sharing nor writability should be assumed from the method name alone.
Request an explicit copy when handing data to a library that mutates arrays. Forcing a shared array’s writeable flag back on can bypass pandas’ protection and undermine the isolation expected elsewhere in the pipeline. Also inspect the resulting dtype: exporting nullable or mixed columns may create an object array rather than a compact numeric view.
82. How would you isolate mutable cell payloads while defining cross-row aliasing?
Define ownership first: should two rows that reference the same list continue to share it, or should each row own an independent payload? A DataFrame deep copy does not recursively clone cell objects, and Copy-on-Write cannot intercept mutation inside them. For independently editable rows, copy the selected payloads individually and test both source isolation and cross-row isolation.
Each call to deepcopy above starts a separate copy operation, deliberately breaking sharing between rows. If shared identity is meaningful, recursively copy the collection of payloads in one operation instead: its memo can preserve shared references within the new graph. Do not assume that calling deepcopy on the DataFrame itself recursively clones cell objects; pandas supplies its own copying behavior.
Apply this policy only to columns that contain such objects, and account for the memory cost. Custom objects can override copying or contain resources that should not be cloned. Normalizing payload members into another table, or using immutable values, often gives a clearer contract than preserving an arbitrary mutable object graph.
83. Why can reindexing fail when duplicate labels are present?
A source label that occurs more than once does not identify a unique value to align into a new index. Operations that need that unique mapping can reject duplicate source labels rather than guessing which row should be used. Duplicate labels are permitted in many pandas objects, so an earlier successful selection does not establish that later alignment is safe.
Resolve the underlying meaning before fixing the index: aggregate repeated observations, retain a separate row identifier, or reject erroneous duplicates. Resetting the index can make an operation run while discarding the identity needed for correct joins. Validate uniqueness at the boundary where the pipeline first depends on it.
84. How does MultiIndex sorting affect range slicing?
MultiIndex range slicing relies on the relevant levels being sorted. A full tuple lookup may work on an unsorted index while a range slice fails or a less selective lookup becomes expensive. Sort the index in the level order that matches the intended access pattern, and measure whether maintaining that order pays off for the workload.
Sorting changes row order, which can matter for first/last operations or later positional assignment. Preserve an explicit sequence column if input order has meaning. An index optimized for customer-then-date access is not automatically optimized for date-then-customer queries.
85. When would you disallow duplicate labels explicitly?
After ingestion cleanup, set_flags(allows_duplicate_labels=False) can make uniqueness an explicit invariant. It rejects an object that already has duplicate labels and can catch transformations that introduce them. This is useful when a downstream lookup assumes one result per label.
The duplicate-label flag is documented as experimental, and propagation is not complete across all methods. Keep explicit index.is_unique and column checks at critical interfaces rather than treating the flag as a database constraint. Apply it after legitimate duplicate raw observations have been handled, not as a substitute for defining their meaning.
86. How can mismatched categorical keys affect a merge?
Categorical values are encoded against a category vocabulary. Two join keys with different vocabularies may lose categorical representation during a merge, changing memory use and dtype expectations even when the matched values look correct. Compatible categories should express a shared domain, rather than merely forcing a smaller storage type.
If the domain truly is shared, harmonize categories before joining and assert the output dtype. Casting to an incomplete vocabulary turns unknown values into missing values, which can then alter matching behavior. Ordered categories require compatible ordering semantics; alphabetically unioning them is not a valid replacement for a business-defined order.
87. How do the str and string dtypes differ in pandas 3.x?
Pandas 3.x infers a dedicated str dtype for ordinary string data; its missing-value convention uses NaN. Explicit dtype="string" retains the nullable string behavior with pd.NA. That difference affects comparison results and downstream Boolean handling, even when displayed values appear similar.
Declare the missing-value contract at input and output boundaries, and test both missing and nonmissing cases during an upgrade. Do not detect strings solely with dtype == object. Storage may use Python or PyArrow depending on configuration and installed dependencies, so semantic dtype choices and physical storage choices deserve separate tests.
88. What should be checked before adopting Arrow-backed pandas dtypes?
Check the operations and interchange boundaries used by the application, not just whether loading a frame succeeds. Arrow-backed dtypes can preserve nullable types and offer efficient storage, but supported operations, conversions and resulting dtypes depend on pandas and PyArrow versions. A NumPy conversion or unsupported operation may allocate new storage or change representation.
Boundary | Useful assertion |
|---|---|
Read | Requested backend and expected nullable dtype are present |
Transform | Values, missingness and output dtype remain acceptable |
Export | The receiving library supports the representation or conversion is explicit |
Use representative strings, timestamps, nulls and large integers in a small round-trip fixture. Pin and test the optional dependency before comparing memory or speed. Arrow availability is not guaranteed by a pandas installation, and choosing an Arrow backend does not automatically make every operation zero-copy or faster.
89. How does nullable Boolean logic affect filtering?
The nullable Boolean dtype uses three-valued logic. An unknown combined with a decisive operand can become known: false AND unknown is false, while true OR unknown is true. When a nullable Boolean mask is used for indexing, missing mask entries are excluded. That last rule is a filtering policy, not evidence that the original condition was false.
For an eligibility report, keep an explicit unknown bucket if missing evidence should trigger review. Filling unknown with true or false is a business decision and should be visible. Using ordinary Python truth testing on pd.NA raises an ambiguity error rather than selecting a default.
90. How can numeric dtype choices silently damage exact identifiers or arithmetic?
Fixed-width numeric storage has finite range, and floating-point storage cannot exactly represent every large integer. Unsigned arithmetic can wrap rather than produce a negative result. A conversion that makes a calculation convenient can therefore break identifier equality or an inventory-balance invariant.
Choose a domain-appropriate type before arithmetic and assert bounds before casting. Nullable integer types preserve missingness but do not provide arbitrary precision. For decimal accounting or integers beyond fixed-width limits, select an explicit representation and verify that joins, storage and external libraries preserve it end to end.
91. Why should floating-point reduction tests use meaningful tolerances?
Floating-point sums can change with reduction order because addition is not exactly associative at finite precision. A different partitioning or backend can produce a small difference without changing the mathematical algorithm. Conversely, a large relative tolerance can hide a material error near a business threshold.
Set absolute and relative tolerances from the units, scale and acceptable error of the result. Near zero, absolute tolerance often dominates; for counts and exact identifiers, approximate equality is usually wrong. Include cancellation-heavy and large-magnitude inputs in important numerical tests, and do not claim a two-value fixture establishes stability for every workload.
92. What migration issue does include_groups address in GroupBy.apply?
In pandas 3.x, grouping columns are excluded from the frame passed into a DataFrameGroupBy apply function; include_groups=True is no longer allowed. Older code that accidentally included a numeric grouping column in an aggregate can therefore change behavior or fail when it expects that column to be present.
Select the input columns explicitly and use built-in aggregation when it expresses the operation. If group identity is needed, design that dependency deliberately rather than assuming the key is a data column in the UDF. Migration fixtures should compare index structure and dtype as well as totals, since apply can reshape the result.
93. How can grouped rolling results be safely assigned back to source rows?
Grouped rolling commonly returns an index containing both the group key and original row index. Dropping a level may be sufficient only when the remaining row identity is unique and matches the source frame. Assigning raw values positionally can silently attach results to different rows because grouped output may be ordered by group.
For real time windows, sort each group in the required time order while retaining an immutable row identifier. Reintegrate using that identifier and assert one result per eligible source row. Duplicate timestamps are not necessarily duplicate events, so replacing row identity with time alone may make the mapping ambiguous.
94. When is a custom BaseIndexer appropriate for rolling calculations?
Use a custom indexer when the window membership rule cannot be expressed by the built-in fixed-size or time-offset windows. Its get_window_bounds method returns integer start and end positions, with end positions exclusive. Those bounds define the data seen by the subsequent reduction.
This minimal indexer deliberately rejects options it does not implement; its expanding behavior is already available through a built-in method. For a real event-reset window, test boundary arrays independently, including empty input and the reset row itself. Incorrect bounds can leak future observations into a historical calculation even when the output shape looks right.
95. When is the Numba engine a reasonable choice for rolling apply?
Consider it for a numeric custom function that cannot be expressed efficiently with a built-in reduction and whose workload is large enough to justify compilation. Rolling apply with the Numba engine requires raw=True, so the function receives NumPy arrays rather than labeled Series. Numba must be installed with compatible dependencies.
Benchmark the first call separately from warmed calls, and compare equivalent missing-value and minimum-period behavior. A compiled function that omits a null policy is not a valid speed comparison against a pandas method that implements one. Parallel execution also has overhead and does not help every window size or function.
Keep a small trusted reference implementation and compare results within appropriate tolerances before measuring speed. If the function needs Python objects or pandas operations unsupported by compilation, rewrite the algorithm or use another execution path; specifying an engine is not a guarantee that arbitrary Python becomes compiled code.
96. How can merge_ordered forward filling change the meaning of a result?
merge_ordered combines ordered key sequences and can forward-fill values introduced by the merge. That may represent a last-known observation, but it can also carry a stale value across a gap. Its output includes the ordered union appropriate to the selected join; it is not simply a nearest-match lookup for each left row.
The carried rate on day 2 is an explicit modeling assumption. Preserve the observation timestamp when age matters and enforce an acceptable staleness policy separately. For multiple entities, ensure filling is confined to the intended entity groups; a global ordered merge can spread information across unrelated series.
97. How would you estimate a many-to-many merge before allocating its result?
For each matching key, an inner many-to-many merge produces the product of the left and right multiplicities. Summing those products estimates the output row count before materializing the full rows. A few highly repeated keys can dominate the result even when most keys are unique.
Include missing-key behavior because pandas matches null keys to null keys, unlike a typical SQL equality join. Multiply the row estimate by a realistic output width and allow headroom for indexers and temporary arrays. If the relationship is unintended, fix the grain; chunking an accidental Cartesian expansion only spreads out the same incorrect work.
98. Why is repeatedly concatenating growing frames inefficient?
Repeatedly concatenating the accumulated result with one more chunk can repeatedly allocate or copy growing arrays. With similarly sized chunks, the cumulative work can approach quadratic growth in total data size. Copy-on-Write does not make every shape-changing concatenation free.
Collect manageable pieces and concatenate once when the final result fits memory. Check column schemas first so alignment does not unexpectedly introduce wide sparse columns or promote dtypes. If the final result itself is too large, write partitions or aggregate incrementally instead of keeping both all chunks and the final assembled frame in memory.
99. Which aggregations can be computed correctly across CSV chunks?
Chunking works when partial state can be combined correctly. Sums and counts compose directly; a mean needs both sum and nonmissing count, not the unweighted average of chunk means. Exact distinct counts and duplicate removal require state that crosses chunk boundaries.
If an identifier appears in two different chunks, local duplicate removal will keep it twice. Maintain a global key set only when its memory is acceptable, or partition by key and resolve duplicates with a defined survivor rule. Sorting, medians and ordered windows require more coordination than summing partial results.
100. How do you detect CSV schema drift without silently losing valid values?
Separate structural parsing errors from semantic type errors and legitimate text values. A line with an unexpected field count is different from an amount containing a currency symbol, and the literal text NA might be a valid code rather than a missing value. Skipping bad lines hides evidence unless rejected rows are reconciled.
Preserve raw lexical values long enough to report conversion failures and enforce an agreed missing-token policy. Validate required columns, unexpected columns and row counts in addition to dtypes. Chunked inference can otherwise give different representations to the same column as new values arrive.
101. What must a Parquet round-trip test verify beyond matching values?
Verify dtypes, nulls, index representation and timezone metadata as well as values. Parquet stores typed columns, but the pandas-to-engine mapping and metadata determine how the frame is reconstructed. The selected optional engine and its version are part of the contract.
Fixture feature | Failure to detect |
|---|---|
Nullable integer with missing values | Conversion to floating point or loss of nullable dtype |
Named nondefault index | Index omitted or reconstructed as an ordinary column |
Timezone-aware timestamps | Instants or timezone metadata changed |
Categorical column | Category vocabulary or ordering lost |
Choose whether the index is meaningful before writing: index=False deliberately omits it, while default behavior can encode a RangeIndex efficiently in metadata. Round-trip through the actual consumer too if another system reads the file. A pandas-only test cannot guarantee that every external reader interprets pandas metadata identically.
102. How do SQL read and write boundaries affect pandas types and transactions?
A SQL table has its own schema, constraints and transaction behavior. to_sql writes the index by default, so specify index=False when row labels are not database data. SQL nulls may also be reconstructed into a different pandas dtype unless the read-side contract is explicit.
This in-memory SQLite example checks values and dtype, not another database’s transaction behavior. With a SQLAlchemy connection already in a transaction, pandas does not commit that transaction itself. The legacy sqlite3 path has different rollback limitations. Test failures with the actual connector, and avoid treating if_exists="replace" as an ordinary append: it drops the existing table before insertion.
103. How does JSON table orientation differ from records for round trips?
orient="records" produces an interoperable list of row objects but does not carry the full pandas dtype and index schema. orient="table" includes a schema describing fields and index information, making it more suitable for supported pandas round trips. Neither choice removes the need to define date precision and missing-value semantics.
For an API, prioritize its documented schema over pandas convenience. Decide whether timestamps are ISO strings or epoch values and specify the unit when using epochs. Test large integers and decimals with the receiving language because its JSON number representation can lose precision even when pandas emitted the intended digits.
104. Why is an Excel round trip not a reliable formula or type-preservation test?
Excel files contain cell types, formatting and sometimes formulas or cached formula results; a DataFrame does not represent all of that workbook behavior. pandas uses optional engines to read and write these files, and it does not calculate Excel formulas. A cached result can be missing or stale until a compatible spreadsheet application recalculates it.
For a monthly workbook, verify that date cells are interpreted as dates rather than formatted text, identifiers retain leading zeroes, and formula handling matches the engine options. A displayed number format does not prove that the underlying value is an exact identifier. Excel also does not natively preserve timezone-aware datetime values as pandas does.
Keep workbook-level requirements separate from tabular-data assertions. If formulas, styles, macros or workbook links matter, use an appropriate workbook-aware tool and test in the target application. For pure data exchange, an explicit CSV or typed columnar schema may make the contract easier to validate, depending on the recipient.
The workbook above contains an unevaluated formula and no cached result. The default pandas read returns a missing value for that cell, while a formula-aware openpyxl read retrieves the formula text. This does not evaluate the formula or establish how a separately recalculated workbook behaves.
105. What does memory_usage(deep=True) omit from a memory investigation?
It estimates the memory attributed to pandas columns and optionally the index, including additional object-dtype accounting. It is not a measurement of the entire process, peak intermediate allocations or a complete recursive object graph. Shared objects and views can also complicate how separate estimates should be added.
Use this estimate to identify expensive columns, then measure process peak memory around representative operations. A merge may need much more memory than either input because of indexers and output expansion. Deleting a DataFrame does not guarantee the operating system immediately observes an equal memory drop, so distinguish object lifetime from allocator behavior.
106. When does a sparse dtype save memory, and when can it cost more?
Sparse storage keeps explicit non-fill values plus their locations. It is attractive when one fill value dominates, but index overhead can outweigh savings as density rises. The fill value is part of the representation: treating zero as implicit is a different contract from treating missing values as implicit.
Inspect density and memory on representative data after transformations, not just at ingestion. An operation can change the fill value or lead to a denser result, and external libraries may require conversion to dense arrays. Reserve enough memory for that boundary rather than assuming sparse storage guarantees a small end-to-end footprint.
107. Are pandas string methods always faster than a Python loop?
No. A vectorized-looking API does not guarantee a native-code fast path for every dtype and operation. Object-backed strings, Python callbacks, regular expressions and intermediate lists can have very different costs. String storage backed by Arrow can also behave differently from Python-backed storage.
For example, splitting every sentence into a list and then counting list lengths creates an intermediate collection that may dominate memory. Benchmark the exact operation against a simple reference using representative string lengths, null rates and dtype. Compare identical semantics: whitespace splitting, literal separators and regex patterns can produce different counts.
Prefer the clearest correct expression first, then profile the actual bottleneck. Measure both runtime and peak memory; a faster method that exceeds the memory budget is not an improvement for the job. Do not generalize a short-string microbenchmark to long documents or claim that all Python loops are necessarily slower.
108. How can iterrows change the type or precision of a row value?
iterrows represents each row as a Series, which can coerce mixed column types into a common dtype. An integer identifier alongside a float can become a float in the row and lose precision at sufficiently large values. itertuples usually preserves scalar types more faithfully and avoids constructing a Series per row.
If row-wise integration with another library is unavoidable, use explicit columns and validate their types before iteration. Do not assign back through the objects returned by iteration and expect the original frame to update. Named tuple field names can also be rewritten for invalid or repeated column names, so positional tuples can be clearer at a controlled interface.
109. What are the performance and security limits of pandas eval?
eval supports a restricted expression language, and an optional numexpr engine can accelerate some sufficiently large numeric expressions while reducing temporaries. It does not accelerate arbitrary Python, and engine compatibility depends on the dtypes and operations involved. Small expressions can lose time to parsing and dispatch overhead.
The explicit Python engine makes this example independent of numexpr availability; it is a correctness example, not a speed benchmark. Never pass untrusted user expressions to eval or query as a supposed sandbox: their documentation warns that they can run arbitrary code. Expose a constrained application-level operation model instead of accepting unrestricted expression strings.
110. Which operations become incorrect when a frame is processed in independent partitions?
Operations requiring global order or cross-partition state cannot be implemented by blindly applying the same local operation everywhere. Global ranking, exact deduplication, joins with keys in several partitions and rolling windows near boundaries all need coordination. More workers do not fix a missing data dependency.
For fixed-size windows, carry an appropriate overlap and discard duplicate boundary outputs. Key-based operations may require repartitioning or exchanging state rather than overlap. Test a fixture with the difficult key or window deliberately split across partitions; random small samples can accidentally place every dependency in one partition and hide the defect.
111. How would you decide whether to move a workload beyond in-memory pandas?
First identify the limiting operation and its data dependencies. Column projection, early filtering, better dtypes or chunked aggregation may make the workload fit. If the irreducible result or intermediate join exceeds memory, choose a system whose execution model supports the required operation and storage pattern.
Workload | Evaluation focus |
|---|---|
Large scan and grouped aggregation | Predicate pushdown, partial aggregation and disk access |
Large key-based join | Partitioning, skew and shuffle costs |
Arbitrary Python row logic | Parallel overhead, serialization and supported execution |
Interactive small result | Latency and cost of moving data to the client |
A pandas-like API in another library does not promise identical ordering, null, categorical or index semantics. Build conformance fixtures for the operations that matter before comparing performance. Include data transfer and materialization in the benchmark; a lazy plan that looks fast to construct has not yet completed the work.
112. Why should hash_pandas_object not be used as a permanent business identifier?
It provides useful hashes for pandas values, including index values by default, but a finite hash is not a uniqueness guarantee or a documented durable cross-system identity scheme. Representation choices and parameters affect what is hashed. Equal hashes are useful candidates for equality checks, not proof that two records are identical.
For a durable content identifier, specify canonical field order, type encoding, missingness and timezone treatment, then use an appropriate versioned hashing protocol. Retain a collision-resolution or full-equality strategy where correctness depends on identity. Do not use pandas hashing as password protection or a cryptographic authenticity check.
113. How should assert_frame_equal express a pipeline’s actual contract?
Select strictness intentionally. Values alone may be insufficient when dtype, column order, index names or categorical metadata matter. Conversely, floating-point outputs may need documented tolerances rather than exact bitwise equality. Use assert_frame_equal to make those decisions explicit.
Do not disable dtype checks simply to make a failing test pass; an integer-to-float conversion may be a real identifier bug. Likewise, check_like=True is appropriate only when row and column ordering are irrelevant to the contract. Keep small fixtures covering empty inputs, missing values and duplicated keys so strictness is tested where it matters.
114. How does NumPy ufunc behavior change when Series are converted to arrays?
Many NumPy ufuncs dispatched on pandas Series preserve pandas behavior, including label alignment between Series. After explicit conversion to NumPy arrays, labels are gone and operations are positional. That boundary can change a numerically valid calculation into the wrong pairing of observations.
If a third-party numeric kernel expects positional arrays, align the inputs explicitly, assert their index equality, and only then convert. Record the intended output index before rebuilding a Series. Also test dtype and missing-value conversion, since extension arrays may require copying or a different representation at the NumPy boundary.
115. What is lost when a timezone-aware timestamp is serialized as a local clock string?
A local clock reading without its offset or zone cannot always identify an instant. During a daylight-saving fall-back transition, two different instants can have the same local wall-clock value. Keeping only a formatted date and time therefore loses information needed to restore the original timeline.
For event exchange, preserve an unambiguous instant, commonly in UTC, and retain the named business timezone separately when local scheduling rules matter. An offset identifies the instant but does not capture a zone’s future daylight-saving rules. Test the actual file format and parser instead of assuming that a displayed timezone survives a round trip.
116. How should idxmax and idxmin handle ties and entirely missing data?
They return the label of the first occurrence of the selected extreme, so ties are resolved by row order rather than a business ranking rule. In pandas 3.1, an all-missing Series raises ValueError; code should define what no eligible observation means before calling the method.
If ties should favor the latest event or lowest cost, encode that rule with explicit ordering and test it. A returned label is not necessarily a unique row when the index contains duplicates. For an all-missing group, return an explicit unavailable status or exclude it according to the reporting contract rather than inventing a winning row.
117. Why is a business-day offset incomplete without a calendar policy?
A weekday rule does not identify local holidays, exceptional closures or a particular market’s operating calendar. CustomBusinessDay can encode a weekmask and holidays, but the caller must supply the correct calendar and keep its coverage current.
This example skips a weekend and one explicitly supplied closure; it does not claim that date is an actual public holiday. Preserve the calendar parameters with the output, since the generic custom-business-day frequency label does not fully describe them. For trading sessions, half-days or cutoff times, a date-only offset may be insufficient and requires a richer scheduling model.
118. How do closed endpoints and overlapping intervals affect IntervalIndex lookups?
An interval’s closure determines whether its left and right endpoints belong to it. With overlapping intervals, a point can match several entries, so get_loc can return a slice or Boolean mask instead of one integer position. A lookup function must not assume that every point identifies exactly one bucket.
For nonoverlapping business bands, validate overlap and test each boundary explicitly. If overlaps represent legitimate concurrent entitlements, preserve all matches rather than selecting the first accidentally. Distinguish point membership from looking up an exact Interval object, which follows different matching semantics.
119. What does infer_freq tell you about an irregular or timezone-aware series?
infer_freq looks for a regular pattern in observed datetime-like values; it does not recover the intended business schedule from incomplete data. It returns no inferred frequency when the pattern is not regular and requires at least three values. A missed observation can therefore break inference even when a scheduled process is nominally daily.
The daily local schedule above includes a 23-hour elapsed interval across daylight-saving time. Decide whether the requirement concerns local calendar days or fixed elapsed durations before converting to UTC or filling gaps. Keep the declared schedule as metadata and compare observations with it; inference alone cannot distinguish a missing run from an intentionally irregular process.
Pandas 3.1 introduces an option for returning an offset object rather than a frequency string. The example opts into that behavior and inspects freqstr; this makes the version-specific return contract explicit instead of relying on a deprecated default.
120. What should a pandas upgrade compatibility matrix include?
Include the pandas version, Python and NumPy versions, relevant optional engines, dtype choices and configured options that affect the application. A pandas upgrade can alter defaults, warnings, dtype inference or backend behavior even when an import succeeds. The dependency combination used in production is the unit that needs verification.
Test group | Representative contract |
|---|---|
Input and output | CSV null tokens, Parquet schema, SQL types and timestamp units |
Core transformations | Index alignment, group result shape and missing-value behavior |
Memory and performance | Representative joins, copies and peak allocation |
Optional acceleration | Equivalent results with supported engine versions |
Run a small edge-case suite and a representative end-to-end fixture in the old and candidate environments, investigate warnings, and compare intentional contract changes explicitly. Pinning provides reproducibility only if the environment can actually be rebuilt, so retain the lock or dependency manifest and interpreter details. Do not relax every equality check to hide a migration regression.
pandas Documentation and Further Reading
These official pandas references cover the APIs and semantics used above. They follow the current documentation, which was version 3.1 when these examples were prepared. Check your installed pandas version, optional engines and dependency versions when reproducing behavior, particularly for Copy-on-Write, string storage and file-format round trips.
Similar Posts
