Back to interview questions

100+ Large Language Models Interview Questions and Answers

100+ Large Language Models Interview Questions and Answers

Explore 120 large language model interview questions and answers on transformer architecture, training objectives, distributed training, caching and inference.

Explore 120 large language model interview questions and answers on transformer architecture, training objectives, distributed training, caching and inference.

By

Team Sherlock.sh

Add Sherlock as a preferred source on Google

Large language model interviews should reveal whether a candidate understands the mechanisms behind fluent output and can turn those mechanisms into a reliable system. Recruiters need to assess more than familiarity with model names or prompting techniques: training objectives, inference costs, data quality and evaluation design all affect what an application can deliver. A plausible demonstration is a starting point, not evidence that a system will remain accurate or dependable in use.

These 100+ Large Language Models Interview Questions and Answers cover 120 questions across beginner, intermediate and experienced levels. The focus is model engineering: token and tensor representations, transformer blocks, training objectives, optimization, distributed execution, cache correctness and inference kernels. The progression moves from tracing a small decoder to diagnosing numerical, synchronization and state-management failures in a larger system.

How can recruiters distinguish an understanding of LLM systems from polished descriptions of their capabilities?

Ask the candidate to annotate tensor shapes, align shifted labels or explain why cached logits differ from an uncached reference. Then change one assumption, such as padding, sequence length or worker count. Strong candidates identify the invariant that should hold, propose a small falsifiable test and distinguish a measured result from a plausible performance claim.

Sherlock AI helps hiring teams assess signals of potential AI cheating, deepfakes, impersonation and unauthorized assistance during remote interviews. Pair those signals with architecture walkthroughs, calculation exercises and follow-up questions about evaluation limits. Use them as context for a fair review alongside demonstrated reasoning, rather than conclusive proof of misconduct.

Large Language Models Interview Questions

Choose questions for model implementation, training infrastructure or inference engineering roles. Ask candidates to state the architecture and numerical assumptions before calculating memory or changing an execution path. The complete bank comes first, with matching numbered answers afterward.

Large Language Models Interview Questions for Beginners

Check whether candidates can explain tokens, probability distributions and the training objective without confusing them with guaranteed knowledge or reasoning. Ask for a concrete example of what a model sees and predicts.

  1. What tensors pass through a decoder-only language model?

  2. Why must token IDs stay paired with the correct embedding matrix?

  3. How does an embedding lookup differ from a one-hot matrix multiplication?

  4. What are logits, and why should cross-entropy usually receive them directly?

  5. Why does stable softmax subtract the largest logit?

  6. How are inputs and targets shifted for next-token training?

  7. Why is a causal mask necessary even when targets are shifted?

  8. What is the difference between an attention mask and a loss mask?

  9. What do BOS, EOS and padding tokens each mean?

  10. Why can left padding matter during batched decoder generation?

  11. What shapes do queries, keys and values have in multi-head attention?

  12. Why are attention scores scaled by the square root of head dimension?

  13. What does concatenating attention heads accomplish?

  14. How do MHA, MQA and GQA differ in their projection structure?

  15. What is a residual connection doing in a transformer block?

  16. How do pre-norm and post-norm transformer blocks differ?

  17. How does RMSNorm differ from LayerNorm?

  18. Why is the feed-forward network applied separately at each token position?

  19. What changes when an MLP uses a gated activation such as SwiGLU?

  20. What does tying input embeddings to the output projection mean?

  21. What is a position ID, and how is it different from a token ID?

  22. How do rotary position embeddings change queries and keys?

  23. How do model parameters, activations and optimizer states differ?

  24. What do forward pass, backward pass and optimizer step each do?

  25. What is a microbatch, and how does it relate to global batch size?

  26. Why can averaging batch losses give the wrong token-level loss?

  27. How does a training epoch differ from a token budget?

  28. What does learning-rate warmup change?

  29. What is gradient clipping, and what does it not fix?

  30. How do FP16 and BF16 differ for language-model computation?

  31. Why does mixed-precision training sometimes use loss scaling?

  32. Why does evaluation mode differ from disabling gradients?

  33. What information belongs in a model checkpoint versus a training checkpoint?

  34. How is prefill computation different from one decode step?

  35. Why does a causal decoder cache keys and values rather than every old attention output?

  36. What is the distinction between allocated cache capacity and valid cache length?

  37. What do greedy decoding and sampling do with a logit vector?

  38. Why should stop tokens and maximum output length be tested separately?

  39. How can a tiny overfit test reveal a broken training pipeline?

  40. What would you test before trusting a custom transformer implementation?

Large Language Models Interview Questions for Intermediate

Explore data packing, objective normalization, adaptation, caching, kernels and quantization. Ask candidates to explain both the mechanism and a concrete test for a plausible implementation error.

  1. How would you verify that packed training examples cannot attend across boundaries?

  2. How should completion-only loss handle multi-turn chat examples?

  3. Why can a tokenizer or chat-template change invalidate a fine-tuning comparison?

  4. How would you compute perplexity over overlapping evaluation windows?

  5. How can truncation silently change a supervised training objective?

  6. What does deduplication change beyond reducing dataset size?

  7. How do data-mixture weights translate into actual training exposure?

  8. How do Adam moments affect a resumed training run?

  9. How does decoupled weight decay differ from adding an L2 term to an adaptive optimizer loss?

  10. When is gradient accumulation equivalent to a larger batch?

  11. What does activation checkpointing trade for memory?

  12. How would you locate the first source of nonfinite values in training?

  13. How does data-parallel gradient synchronization work?

  14. What is the difference between sharding optimizer states, gradients and parameters?

  15. How should a learning-rate schedule behave when an update is skipped?

  16. Why is one LoRA matrix often initialized to zero?

  17. How would you choose LoRA target modules without treating rank as the only capacity control?

  18. Why can merging an adapter before quantization differ from merging after quantization?

  19. How is a completion log-probability calculated for preference training?

  20. What does the reference-policy ratio contribute to the basic DPO objective?

  21. How would you test a KV cache against a full-sequence reference?

  22. Why can a square causal mask be wrong for cached multi-token decoding?

  23. What happens to a KV cache when beam search reorders hypotheses?

  24. How does sliding-window attention change cache retention?

  25. Why does FlashAttention avoid storing the full attention matrix?

  26. How does an online softmax combine two blocks of scores?

  27. What does a paged KV-cache block table represent?

  28. Why does sharing a prefix require copy-on-write behavior when sequences branch?

  29. How does chunked prefill change interference with active decoding?

  30. Why can a static cache help compilation while wasting memory?

  31. What is the difference between weight-only and activation quantization?

  32. How do scale and zero point define affine quantization?

  33. Why can per-channel quantization preserve more useful detail than one global scale?

  34. How would you detect calibration data that is unsuitable for activation quantization?

  35. Why must speculative sampling use both draft and target probabilities?

  36. How should a speculative decoder handle a rejected draft suffix?

  37. How would you benchmark a GPU kernel without timing only its launch?

  38. What is arithmetic intensity, and why does it matter for decoding?

  39. Why can vocabulary projection and sampling become significant after optimizing attention?

  40. What evidence should accompany a claim that an inference optimization preserves behavior?

Large Language Models Interview Questions for Experienced

Probe experimental design, distributed execution and reliability boundaries. Ask what would falsify an architectural claim, invalidate a benchmark or require a deployment to be rolled back.

  1. How would you derive token-correct gradients when distributed ranks have unequal target counts?

  2. How can a distributed training job hang even when every GPU has enough memory?

  3. What makes exact training resume difficult when the number of workers changes?

  4. How would you prove a checkpoint is a consistent snapshot during asynchronous saving?

  5. How do pipeline schedules trade activation memory against idle time?

  6. Why can tensor-parallel performance collapse across a slow interconnect?

  7. What does sequence parallelism shard that ordinary tensor parallelism may leave replicated?

  8. How would you diagnose a mixture-of-experts model whose experts receive highly uneven traffic?

  9. Why do active parameters not fully predict the cost of a mixture-of-experts model?

  10. How would you test that an all-to-all expert dispatcher preserves token identity?

  11. Why can model FLOPs utilization be misleading when comparing training systems?

  12. How would you design a controlled ablation of a new attention architecture?

  13. What evidence is needed before extrapolating a scaling law to a larger run?

  14. How would you distinguish undertraining from an optimization failure?

  15. How should continued pretraining be evaluated for catastrophic forgetting?

  16. How can a preference dataset accidentally reward response length instead of quality?

  17. How would you diagnose reward hacking in an online language-model training loop?

  18. What makes distillation from teacher logits different from training on teacher-generated text?

  19. How would you distinguish benchmark contamination from legitimate generalization?

  20. How would you test long-context positional scaling independently of retrieval difficulty?

  21. Why can a RoPE cache become invalid after changing the positional scaling configuration?

  22. How would you validate a quantized KV cache separately from quantized weights?

  23. What correctness risks arise when reusing CUDA graphs for dynamic request batches?

  24. How do you reason about prefill/decode disaggregation without ignoring cache transfer?

  25. How would you make cache eviction and request cancellation safe under concurrent kernels?

  26. Why can identical seeded requests differ when batching changes?

  27. How would you validate speculative decoding when a grammar constrains allowed tokens?

  28. What can go wrong when temperature and top-p are applied in different orders?

  29. Why can an all-masked attention row produce NaNs, and what should define its behavior?

  30. How would you compare attention kernels when tiny logit differences become large generation differences?

  31. How can an inference memory leak be distinguished from intentional allocator caching?

  32. How would you diagnose performance regressions caused by dynamic-shape recompilation?

  33. Why can a faster tokenizer fail to improve end-to-end training throughput?

  34. How should a training data loader resume without silently repeating or skipping records?

  35. What makes tokenizer-vocabulary expansion a model migration?

  36. How would you estimate whether adapter multiplexing is better than separate merged replicas?

  37. How can a model conversion preserve tensor shapes while corrupting attention?

  38. What should a minimal reference implementation sacrifice to remain useful?

  39. How would you prioritize tests for a new distributed language-model training stack?

  40. What would make you reject an apparently successful LLM optimization experiment?

Large Language Models Interview Questions and Answers

The answers primarily use autoregressive transformer decoders and identify implementation-dependent behavior. Python examples use only the standard library to illustrate arithmetic and indexing; they do not train a model or benchmark an accelerator. Framework APIs, kernel support and checkpoint conventions must be checked against the installed versions.

Large Language Models Interview Questions and Answers for Beginners

These answers establish the core architecture and vocabulary, with examples that separate mathematical definitions from broader capability claims.

1. What tensors pass through a decoder-only language model?

Integer token IDs enter an embedding lookup, producing vectors with shape batch × sequence × hidden width. Transformer blocks update those vectors, and an output projection produces batch × sequence × vocabulary logits. A decoding rule selects the next token from the final relevant position.

For a batch of two sequences of length eight, hidden width 64 and vocabulary 1,000, the embedding output has shape 2 × 8 × 64 and the logits have shape 2 × 8 × 1,000. The vocabulary axis represents alternative tokens; it is not an extra time axis.

I would annotate these shapes before debugging the model. Accidentally selecting the vocabulary axis as the sequence axis can yield valid-looking tensors with meaningless predictions. The example describes a conventional causal decoder; other architectures can expose different intermediate structures.

2. Why must token IDs stay paired with the correct embedding matrix?

An ID is an index into a learned embedding table. Its numeric value has no universal meaning across tokenizers. If one tokenizer maps ID 42 to a space and another maps it to a word fragment, feeding the second tokenizer into the first model changes the input semantics.

A compatible tokenizer bundle includes vocabulary, normalization, segmentation rules and special-token assignments. Keeping only a vocabulary size is insufficient: two equally sized vocabularies can have entirely different mappings.

I would version the tokenizer with the weights and verify known text-to-ID fixtures when loading an artifact. Adding tokens requires resizing and appropriately training their embeddings; silently replacing the tokenizer is not a harmless preprocessing change.

3. How does an embedding lookup differ from a one-hot matrix multiplication?

For a fixed embedding matrix, selecting its row for a token is mathematically equivalent to multiplying that matrix by the corresponding one-hot representation, with orientation chosen consistently. The lookup avoids materializing a mostly zero vector over the entire vocabulary.

With vocabulary size 50,000 and hidden width 1,024, a token needs one integer ID rather than 50,000 explicit one-hot entries. The learned table still stores 50,000 × 1,024 values; lookup efficiency does not remove that parameter cost.

During training, gradients flow to the selected embedding rows, subject to implementation details such as sparse versus dense gradients. Token IDs themselves are discrete inputs, so ordinary backpropagation does not optimize their integer values.

4. What are logits, and why should cross-entropy usually receive them directly?

Logits are unnormalized scores for vocabulary entries. Softmax converts them into probabilities, but a combined cross-entropy implementation typically performs a numerically stable log-softmax internally. Passing already-softmaxed probabilities into an interface expecting logits applies the wrong computation.

For scores 2, 1 and 0, softmax favors the first token but still assigns mass to the others. Adding the same constant to every score leaves those probabilities unchanged; only relative scores matter.

I would inspect the loss function contract and tensor axes. In a language model, the target is usually an integer vocabulary index for each supervised position. A probability-distribution target is a different supported mode in some libraries and must not be confused with an ordinary token-ID target.

5. Why does stable softmax subtract the largest logit?

Subtracting the maximum preserves softmax probabilities while keeping every exponent at most one. This reduces overflow risk when logits are large. It does not repair NaN inputs, an all-masked row or every possible low-precision error.

For logits 1,000 and 999, directly computing exp(1,000) can overflow. Computing exp(0) and exp(-1), then normalizing, gives the same mathematical distribution with manageable intermediate values.

from math import exp, isclose
scores = [1000.0, 999.0]
m = max(scores)
weights = [exp(x - m) for x in scores]
probs = [w / sum(weights) for w in weights]
assert isclose(probs[0], 0.7310585786300049)
assert isclose(sum(probs), 1.0)
print([round(p, 6) for p in probs])  # [0.731059, 0.268941]
from math import exp, isclose
scores = [1000.0, 999.0]
m = max(scores)
weights = [exp(x - m) for x in scores]
probs = [w / sum(weights) for w in weights]
assert isclose(probs[0], 0.7310585786300049)
assert isclose(sum(probs), 1.0)
print([round(p, 6) for p in probs])  # [0.731059, 0.268941]
from math import exp, isclose
scores = [1000.0, 999.0]
m = max(scores)
weights = [exp(x - m) for x in scores]
probs = [w / sum(weights) for w in weights]
assert isclose(probs[0], 0.7310585786300049)
assert isclose(sum(probs), 1.0)
print([round(p, 6) for p in probs])  # [0.731059, 0.268941]

I would use a tested fused implementation in real training. A hand-written example is useful for understanding the invariant, but production kernels also handle dtype, reduction order and memory access. Compare numerical tolerances rather than demand identical floating-point bytes across kernels.

6. How are inputs and targets shifted for next-token training?

The representation at position t predicts the token at position t+1. A simple sequence [BOS, A, B, EOS] therefore pairs inputs [BOS, A, B] with targets [A, B, EOS]. The model must not use B as input to the same prediction that is scored against B.

Some model classes accept unshifted labels and perform the shift inside their loss implementation. Shifting again in the data loader would train the wrong offset. Conversely, a custom forward pass may require the caller to shift explicitly.

tokens = [0, 11, 12, 1]  # illustrative BOS, A, B, EOS IDs
inputs, targets = tokens[:-1], tokens[1:]
assert list(zip(inputs, targets)) == [(0, 11), (11, 12), (12, 1)]
print(list(zip(inputs, targets)))
tokens = [0, 11, 12, 1]  # illustrative BOS, A, B, EOS IDs
inputs, targets = tokens[:-1], tokens[1:]
assert list(zip(inputs, targets)) == [(0, 11), (11, 12), (12, 1)]
print(list(zip(inputs, targets)))
tokens = [0, 11, 12, 1]  # illustrative BOS, A, B, EOS IDs
inputs, targets = tokens[:-1], tokens[1:]
assert list(zip(inputs, targets)) == [(0, 11), (11, 12), (12, 1)]
print(list(zip(inputs, targets)))

I would print one tiny tokenized example with the aligned input, target and loss mask, then compare a hand-calculated loss. This catches off-by-one errors that can otherwise produce a steadily decreasing but misleading training curve.

7. Why is a causal mask necessary even when targets are shifted?

Shifting labels aligns the prediction task; the causal mask controls information flow inside attention. Without the mask, the hidden state at an earlier input position could attend to a later position containing the token it is supposed to predict.

For inputs [BOS, A, B], the BOS position must not attend to A or B when predicting A. The mask normally allows the diagonal because the current input token is legitimate context for predicting the next token.

A useful test changes a future token and verifies that earlier logits remain unchanged in evaluation mode. That checks the behavior of the whole attention implementation rather than merely inspecting a triangular array in the data pipeline.

8. What is the difference between an attention mask and a loss mask?

An attention mask determines which positions can contribute information to other positions. A loss mask determines which predictions contribute to the optimization objective. Ignoring a target in the loss does not prevent its input representation from influencing later tokens.

Mask

Controls

Prompt-only training example

Attention mask

Information visible to each query

Answer sees prompt; padding is excluded

Loss mask

Targets contributing to objective

Prompt targets ignored; selected assistant targets counted

In completion-only training, prompt tokens usually remain visible to the answer while their own prediction losses are ignored. Padding tokens may require both attention exclusion and ignored targets, depending on layout and model implementation.

I would keep these contracts separate in tests. If the answer cannot attend to the prompt, it cannot learn the intended conditional task. If ignored prompt targets are accidentally counted in the denominator, reported loss and gradient scale can change despite apparently correct target masking.

9. What do BOS, EOS and padding tokens each mean?

BOS marks a beginning, EOS marks a learned sequence or turn ending, and padding fills batch positions that are not actual sequence content. Their roles differ even when an implementation reuses one ID for more than one purpose.

If EOS is also used for padding, a mask based only on token value may accidentally remove real EOS targets. That can teach a model to continue instead of stop. The original sequence lengths or explicit masks distinguish real endings from padding.

I would inspect the model-specific template and generation stopping configuration. Adding an extra BOS or omitting a turn-ending token can change behavior without causing a shape error. Special-token handling belongs to the versioned model contract.

10. Why can left padding matter during batched decoder generation?

Many decoder generation loops obtain the next-token logits from the last sequence position in each batch row. With right padding, that position may be padding for shorter prompts. Left padding places each real prompt end at the final position, provided masks and position IDs are handled correctly.

This is an implementation convention rather than a universal requirement of transformers. A loop that gathers the last valid position explicitly can support other layouts. Training collators and generation loops may use different padding choices.

I would compare a short prompt alone with the same prompt beside a longer one. Unexpected changes can expose padding, mask or position bugs. Do not assume that setting a padding side automatically fixes an incompatible model or custom cache implementation.

11. What shapes do queries, keys and values have in multi-head attention?

In conventional multi-head attention, projections produce queries, keys and values that are reshaped into batch × heads × sequence × head dimension. Multiplying Q by the transposed final axes of K produces a score matrix over query and key positions.

For two batches, four heads, eight query positions and head width 16, Q has shape 2 × 4 × 8 × 16. If ten key positions are cached, the score shape is 2 × 4 × 8 × 10, not necessarily square.

I would distinguish head axes from sequence axes before transposing or reshaping. A tensor can have the expected total element count and still mix token positions across heads. Grouped-query attention additionally separates query-head count from KV-head count.

12. Why are attention scores scaled by the square root of head dimension?

Under a simplified independent, unit-variance component model, a query-key dot product has variance proportional to head dimension. Dividing by its square root controls score scale and reduces softmax saturation as the head width grows.

For head dimension 64, the conventional divisor is eight. This is a scale argument, not a statement that every learned query and key actually has unit variance or that every architecture must use the same normalization.

Large score magnitudes can concentrate softmax on a few keys and change gradients. I would check query/key normalization and the specific attention formulation before adding another scale factor; applying the divisor twice changes the model rather than improving numerical stability.

13. What does concatenating attention heads accomplish?

Each head produces a representation using its own attention pattern and value projection. Concatenating head outputs restores a combined feature dimension, which an output projection mixes back into the residual-stream width.

Four heads of width 16 produce 64 concatenated features per token. The concatenation is along the feature axis, not the sequence axis: it does not turn eight tokens into 32 tokens.

The heads need not correspond to clean human labels such as grammar or facts. I would describe them as learned computational components and test any interpretability claim separately. The output projection also means that downstream behavior depends on combinations of heads, not isolated attention maps alone.

14. How do MHA, MQA and GQA differ in their projection structure?

Multi-head attention normally gives each query head its own key and value head. Multi-query attention shares one KV head across query heads. Grouped-query attention shares a KV head within each group of query heads, placing it between those extremes.

Attention form

Query heads (example)

KV heads (example)

Query heads per KV head

MHA

32

32

1

GQA

32

8

4

MQA

32

1

32

With 32 query heads and eight KV heads, each KV head serves four query heads. This reduces KV projections and stored cache relative to 32 KV heads, while preserving 32 query heads. It does not divide every parameter or every operation by four.

The grouping is part of the trained architecture. Repeating KV tensors to satisfy a kernel interface may preserve the mathematical operation but lose some memory benefit. Changing a checkpoint from MHA to GQA requires an explicit conversion and adaptation method, not merely editing a configuration number.

15. What is a residual connection doing in a transformer block?

A residual connection adds a sublayer update to the existing hidden representation. In a simple pre-normalized block, an attention sublayer can be written x + Attention(Norm(x)), followed by another residual update from the feed-forward network.

The addition requires compatible shapes and maintains a path through which information and gradients can flow. It does not mean that the layer is optional or that its update is guaranteed to be small.

I would check residual placement, scaling and dtype when reproducing a checkpoint. Moving normalization across an addition changes the function. A model may load successfully yet give different outputs if the implementation uses the wrong block ordering.

16. How do pre-norm and post-norm transformer blocks differ?

Pre-norm applies normalization before a sublayer and then adds its output to the residual stream. Post-norm applies normalization after the residual addition. These arrangements change both the forward computation and how gradients propagate through depth.

Compare x + F(Norm(x)) with Norm(x + F(x)). They are not algebraically interchangeable. A final normalization can also appear outside the repeated blocks, so a complete implementation must follow the checkpoint architecture.

Pre-norm is often chosen for optimization behavior in deep models, but I would not claim it makes learning-rate tuning or initialization irrelevant. Reproducing a model requires matching normalization placement, epsilon and learned scale parameters rather than choosing the arrangement that sounds more stable.

17. How does RMSNorm differ from LayerNorm?

RMSNorm divides a vector by its root-mean-square magnitude, with an epsilon and learned scale. Standard LayerNorm also subtracts the mean before normalizing variance, and commonly includes learned scale and bias. RMSNorm therefore does not generally make the output mean zero.

For [3, 4], the RMS is sqrt(12.5), so normalization preserves the positive offset while changing magnitude. Mean-centering the same vector first would instead produce values symmetric around zero.

from math import sqrt, isclose
x = [3.0, 4.0]
epsilon = 1e-6
rms = sqrt(sum(v*v for v in x) / len(x) + epsilon)
y = [v / rms for v in x]  # unit learned scale
assert not isclose(sum(y) / len(y), 0.0)
print([round(v, 4) for v in y])  # [0.8485, 1.1314]
from math import sqrt, isclose
x = [3.0, 4.0]
epsilon = 1e-6
rms = sqrt(sum(v*v for v in x) / len(x) + epsilon)
y = [v / rms for v in x]  # unit learned scale
assert not isclose(sum(y) / len(y), 0.0)
print([round(v, 4) for v in y])  # [0.8485, 1.1314]
from math import sqrt, isclose
x = [3.0, 4.0]
epsilon = 1e-6
rms = sqrt(sum(v*v for v in x) / len(x) + epsilon)
y = [v / rms for v in x]  # unit learned scale
assert not isclose(sum(y) / len(y), 0.0)
print([round(v, 4) for v in y])  # [0.8485, 1.1314]

I would match the implementation’s reduction precision and epsilon. Computing squares in a narrow dtype can behave differently from accumulating in float32. The conceptual distinction is useful for architecture literacy; actual performance and stability require measurements in the target implementation.

18. Why is the feed-forward network applied separately at each token position?

A conventional transformer feed-forward sublayer transforms each token’s feature vector using shared weights. Attention mixes information across positions; the feed-forward network then performs nonlinear feature processing at each position.

The usual two-layer form expands hidden width to an intermediate width, applies an activation, and projects back. A batch and sequence dimension can be flattened for efficient matrix multiplication without changing the fact that the same transformation is applied at every position.

This distinction helps locate bugs: accidentally mixing the sequence axis into the feature axis changes the model. It also explains why feed-forward parameter and compute costs can remain substantial even when attention memory is optimized.

19. What changes when an MLP uses a gated activation such as SwiGLU?

A gated MLP forms two projected feature streams and multiplies one by an activated version of the other before projecting back. In a common SwiGLU formulation, the intermediate is SiLU(xWg) multiplied elementwise by xWu, followed by Wd.

Unlike a two-projection MLP, this uses three projection matrices. To compare parameter budgets fairly, the intermediate width may be adjusted; keeping the same width can increase parameters and arithmetic.

I would verify projection orientation and activation choice against the model configuration. A gate is a learned modulation of features, not a hard binary decision, and replacing it with a ReLU MLP changes checkpoint behavior even when input and output shapes match.

20. What does tying input embeddings to the output projection mean?

Weight tying reuses the token embedding matrix for the vocabulary output projection, usually through its transpose under the appropriate convention. It reduces the number of independently learned parameters and connects input and output token representations.

A vocabulary of 50,000 with hidden width 1,024 contains 51.2 million embedding values. Reusing that table can avoid a second equally sized matrix, although additional biases or architecture-specific components may remain.

When adding vocabulary entries or saving a checkpoint, I would preserve the shared-weight relationship. Two tensors with initially identical values are not necessarily tied: separate parameter objects can diverge during training. Some architectures deliberately use untied weights, so this is a model choice rather than a universal optimization.

21. What is a position ID, and how is it different from a token ID?

A token ID selects vocabulary content; a position ID identifies where that token sits for a position-dependent computation. The same word fragment can appear at several positions and retain its token ID while receiving different positional treatment.

With left padding or a reused cache, array index and semantic position need not be identical. A newly generated token must receive the position consistent with the cached prefix, not automatically position zero because its input tensor contains only one token.

I would test position handling with nonempty prefixes and batches of unequal lengths. Incorrect positions often leave shapes valid and may only become obvious as generation grows. The exact convention depends on learned positions, rotary embeddings or another positional method.

22. How do rotary position embeddings change queries and keys?

RoPE applies position-dependent rotations to paired query and key coordinates. The dot product between rotated vectors incorporates relative positional structure because the rotations compose according to the difference between positions.

The operation typically affects attention queries and keys rather than adding a position vector to every hidden state. Pairing layout, rotary dimension and frequency convention must match the trained model.

I would not infer unlimited context quality from the ability to compute rotations at larger indices. Extrapolation changes the positional patterns the model encounters, and scaling methods need their own evaluation. A correct rotation implementation and a useful long-context model are separate claims.

23. How do model parameters, activations and optimizer states differ?

Parameters are learned weights retained between examples. Activations are intermediate values produced while processing a batch. Optimizer states, such as Adam moments, summarize update history and persist across training steps.

Category

Persists across steps?

Typical dependency

Parameters

Yes

Architecture and vocabulary

Saved activations

Only until no longer needed for backward

Batch, length and checkpointing

Optimizer moments

Yes

Trainable parameter set and optimizer

Gradients

Until cleared or consumed

Trainable parameters and accumulation

A model can fit for inference but fail during training because training adds gradients, saved activations and optimizer states. Increasing sequence length mainly changes activation and attention-related costs; it does not add new dense weight matrices.

I would separate these categories in a memory estimate and include temporary buffers and allocator behavior. Counting only parameters times bytes per weight is useful as a lower-bound calculation, not as a complete training-memory requirement.

24. What do forward pass, backward pass and optimizer step each do?

The forward pass computes predictions and a loss. The backward pass computes gradients of that loss with respect to trainable parameters. The optimizer step uses those gradients and its update rule to modify parameters.

Gradient accumulation performs several forward/backward passes before one optimizer step. Without clearing or intentionally accumulating gradients, a new backward pass may add to old gradients rather than replace them.

I would check the ordering of zeroing, backward, clipping and stepping in a minimal training loop. Calling backward does not by itself update weights, and calling the optimizer with stale gradients can change weights without using the intended latest batch.

25. What is a microbatch, and how does it relate to global batch size?

A microbatch is the amount processed in one local forward/backward pass. For ordinary synchronous data parallel training, the effective example count per optimizer step is local microbatch size × accumulation steps × data-parallel replicas, assuming equal batches and no special weighting.

With two examples per replica, eight accumulation steps and four data-parallel replicas, one update uses 64 examples. Tensor-parallel devices sharing one replica do not each add independent training examples to that formula.

For language modeling, supervised token count is often more informative than example count because lengths vary. I would record both and verify loss normalization; 64 short examples and 64 long examples need not produce the same weighting or compute.

26. Why can averaging batch losses give the wrong token-level loss?

If batches contain different numbers of supervised tokens, their mean losses should not receive equal weight when reporting a corpus-wide token mean. Sum the negative log-likelihood contributions and divide by the total eligible token count.

A two-token batch with mean loss 1 and an eight-token batch with mean loss 3 have token-weighted mean (2 × 1 + 8 × 3) / 10 = 2.6. Averaging the two means gives 2, which answers a different question.

from math import isclose
counts, means = [2, 8], [1.0, 3.0]
weighted = sum(n*m for n, m in zip(counts, means)) / sum(counts)
assert isclose(weighted, 2.6)
assert sum(means) / len(means) == 2.0
print(weighted)  # 2.6
from math import isclose
counts, means = [2, 8], [1.0, 3.0]
weighted = sum(n*m for n, m in zip(counts, means)) / sum(counts)
assert isclose(weighted, 2.6)
assert sum(means) / len(means) == 2.0
print(weighted)  # 2.6
from math import isclose
counts, means = [2, 8], [1.0, 3.0]
weighted = sum(n*m for n, m in zip(counts, means)) / sum(counts)
assert isclose(weighted, 2.6)
assert sum(means) / len(means) == 2.0
print(weighted)  # 2.6

The same issue affects gradient accumulation when each microbatch mean is scaled equally. Decide whether the intended objective weights tokens, examples or conversations, then implement that weighting explicitly and exclude ignored targets from its denominator.

27. How does a training epoch differ from a token budget?

An epoch is one traversal of a defined dataset or sampling schedule. A token budget counts the tokens consumed or supervised. In streaming, repeated or resampled corpora, an epoch can be ambiguous unless the dataset boundary is fixed.

For two datasets with the same number of rows, one may contain ten times as many tokens. Comparing runs only by epochs can therefore conceal a large difference in compute and learning signal.

I would report tokenizer, sampling mixture, consumed tokens and supervised tokens alongside update count. Repeated tokens are still compute, but they are not additional unique data. This makes a training comparison reproducible without implying that every token contributes equally to learning.

28. What does learning-rate warmup change?

Warmup gradually increases the learning rate from a smaller value to the intended training level during an initial interval. It changes update magnitude during early optimization, when optimizer statistics and activations may be poorly calibrated.

A schedule can count optimizer steps or tokens. If gradient accumulation changes, a warmup specified as 1,000 updates may correspond to a different amount of data than before. Scheduler calls must follow the chosen unit.

Warmup is not a repair for a wrong mask, corrupted labels or nonfinite inputs. I would first verify the pipeline, then compare schedules under a controlled budget. Record the full schedule and resume position so a restarted run does not accidentally repeat the initial ramp.

29. What is gradient clipping, and what does it not fix?

Gradient clipping limits an update-related quantity, commonly the global gradient norm before the optimizer step. Norm clipping scales all participating gradients together when their combined norm exceeds a threshold, preserving their direction in that operation.

It can reduce the effect of unusually large gradients. It does not make NaN gradients finite, correct a mislabeled dataset or guarantee stable training. A persistently clipped run may indicate an unsuitable learning rate or another problem.

With mixed-precision loss scaling, gradients must be unscaled before applying the intended norm threshold. In distributed or sharded training, the norm must represent the intended global parameter set rather than an arbitrary local shard.

30. How do FP16 and BF16 differ for language-model computation?

Both use 16 bits per value, but they allocate exponent and fraction bits differently. BF16 retains a wider exponent range, similar to float32, with fewer fraction bits than FP16. FP16 provides more fractional precision within its narrower range.

This makes overflow and underflow behavior different even though storage size is equal. Mixed-precision training may retain some reductions, master weights or optimizer state at higher precision; a BF16 run is not necessarily BF16 arithmetic everywhere.

I would verify hardware support and the actual dtype policy rather than choose solely from the checkpoint file size. Lower precision can improve performance, but correctness comparisons need tolerances and task tests, especially around large activations or small gradient values.

31. Why does mixed-precision training sometimes use loss scaling?

Loss scaling multiplies the loss before backpropagation so small gradients are less likely to underflow in a narrow representation. Before the optimizer update, gradients are unscaled so the intended mathematical update is recovered when values remain finite.

Dynamic scaling can reduce the scale after detecting overflow and skip an invalid update. The scaler state is therefore part of resumable training state. A skipped update also needs consistent handling in the learning-rate schedule.

Loss scaling is particularly associated with FP16; BF16’s wider exponent range often changes the need for it. I would inspect the framework policy rather than apply a scaler universally, and monitor skipped updates instead of interpreting all apparent training steps as successful weight updates.

32. Why does evaluation mode differ from disabling gradients?

Evaluation mode changes behavior of modules such as dropout. Disabling gradient recording changes autograd bookkeeping and memory use. One does not automatically imply the other in common frameworks.

A model can run in evaluation mode while gradients are recorded, which is useful for some analyses. It can also run without gradients while dropout remains active if training mode was not changed, leading to variable outputs.

For ordinary deterministic comparison of two implementations, I would disable training-specific stochastic behavior and gradient recording, then control seeds and tolerances as appropriate. These settings do not guarantee identical results across hardware or kernels, but they remove two avoidable confounders.

33. What information belongs in a model checkpoint versus a training checkpoint?

A model checkpoint contains enough parameters and configuration to reconstruct the model. A training checkpoint additionally needs optimizer, scheduler and other state required to continue the update process consistently.

For exact continuation, this can include random-number generators, loss-scaler state, data-loader position and the accumulation boundary. Saving only weights restarts optimization history even if the model’s initial predictions are identical.

I would label checkpoints by intended use and test a save/load round trip. A portable inference artifact and an exact distributed-training resume snapshot have different requirements; treating them as interchangeable can silently change a resumed experiment.

34. How is prefill computation different from one decode step?

Prefill processes the known prompt positions and produces the state needed to begin generation. A cached decode step usually processes one new token per active sequence while attending to previously stored keys and values.

Prefill therefore exposes substantial parallel work across prompt positions. Decoding has a sequential dependency between generated tokens, although many independent sequences can still be batched. The two phases can exercise hardware differently.

I would avoid claiming a universal compute-bound or bandwidth-bound classification without workload measurements. Model dimensions, batch size, attention architecture and sequence length matter. Separating phases is the first step toward understanding which operations are expensive, not a complete performance diagnosis.

35. Why does a causal decoder cache keys and values rather than every old attention output?

At a new token, each layer needs previous keys and values to compute the new query’s attention result. In a standard causal decoder, earlier token representations do not depend on future tokens, so their projected keys and values remain reusable.

Old attention outputs are not a substitute: the new query assigns its own weights to the stored values. The cache preserves inputs to that new weighted combination, not a single reusable summary of all possible future queries.

This reasoning assumes unchanged model parameters, positions and relevant configuration. Reusing a cache after changing an adapter or editing an earlier token can be invalid. Cache correctness should be tested against an uncached forward pass on the same complete sequence.

36. What is the distinction between allocated cache capacity and valid cache length?

Allocated capacity is memory reserved for possible tokens; valid length is the portion containing keys and values that belong to the current sequence. A static cache can reserve far more space than has been written.

Attention must exclude unused capacity. Otherwise zeros or stale values can receive probability mass and alter the result. Updating the write pointer without a consistent mask can produce incorrect output even though the allocation itself is large enough.

I would test empty, partial and full caches, including reuse after a shorter request. Capacity planning and semantic validity are separate: having space for 8,192 tokens does not mean all 8,192 positions are legitimate context.

37. What do greedy decoding and sampling do with a logit vector?

Greedy decoding selects an entry with the greatest score. Sampling draws according to a probability distribution derived from the scores, possibly after configured transformations. These are selection procedures applied after the model computes logits.

A sampler needs a random source and a well-defined treatment of filtered tokens. Greedy selection can still vary across implementations if near-tied scores change numerically, even though it does not intentionally sample.

I would compare logits separately from selected tokens when debugging. Two implementations can have small acceptable score differences yet choose different tokens near a tie, after which their contexts diverge. Conversely, identical first tokens do not establish identical distributions.

38. Why should stop tokens and maximum output length be tested separately?

A stop token is a model-generated signal recognized by the generation loop. A maximum output length is an external bound that ends generation even if the model has not produced a natural ending.

If the stop ID is wrong, a model may keep emitting turn markers until it reaches the cap. If the cap is too small, a correct model may be cut off halfway through a structured answer. These failures require different fixes.

I would test the exact tokenizer IDs, multi-token stop rules where supported, and whether returned output includes the ending marker. Length-limited completion should be reported distinctly from a normal stop so downstream code can detect truncation.

39. How can a tiny overfit test reveal a broken training pipeline?

A tiny overfit test trains on a very small fixed dataset and checks whether the model can drive its loss down and reproduce the intended targets. It is a diagnostic for optimization and data plumbing, not evidence of generalization.

If a few consistent examples cannot be learned, inspect label shifts, masks, trainable parameters, learning rate and optimizer calls before spending a large training budget. Comparing gradients and parameter changes can isolate where the update path is broken.

Passing the test only establishes that this limited setup can fit these examples. Keep it separate from held-out evaluation, and avoid selecting a production model by memorization ability. Restore normal data splits and regularization after the diagnostic.

40. What would you test before trusting a custom transformer implementation?

I would test tensor shapes, causal isolation, masking, numerical finiteness and checkpoint compatibility before judging generated prose. Fluent output is a weak debugging signal because an incorrect model can still emit plausible common tokens.

Useful fixtures compare full-sequence and cached logits, a prompt alone and in a padded batch, and a small reference attention calculation. A save/load round trip should preserve outputs within the stated numerical tolerance.

Then I would test gradients on a small model and run a bounded training diagnostic. These checks establish specific invariants; they do not prove convergence at scale or performance on every device. Each test should identify the failure it can detect and the configuration it actually exercised.

Large Language Models Interview Questions and Answers for Intermediate

These answers connect model mechanisms to system behavior and expose trade-offs through small calculations and realistic failure cases.

41. How would you verify that packed training examples cannot attend across boundaries?

I would build the attention mask from example boundaries and test independence by changing one packed example while checking logits in another. A plain causal mask only blocks future positions; it still permits later packed examples to read earlier ones.

If independent-example semantics are intended, use a block-diagonal causal structure or the equivalent variable-length kernel metadata. Resetting position IDs alone does not prevent cross-example attention. Conversely, continuous-document pretraining may intentionally retain some cross-boundary context.

The test must match the intended objective. Compare packed and separately processed examples with dropout disabled and identical positional conventions. A throughput improvement is not valid if packing silently changes which information the model can use.

42. How should completion-only loss handle multi-turn chat examples?

The loss mask should select the assistant tokens intended as targets while preserving the full permitted conversation as context. A single split at the first assistant response can be wrong when user and assistant turns alternate later in the sequence.

I would inspect token-level role boundaries after applying the actual chat template. Text searches for a role name can match ordinary content and may fail when control tokens or whitespace change. End-of-turn markers also need an explicit inclusion policy.

Test empty answers, multiple assistant turns, tool exchanges and truncation. A row with zero supervised tokens needs deliberate handling rather than a division by zero or a misleading zero loss. The mask should be derived from reliable formatting metadata or a validated tokenizer-specific procedure.

43. Why can a tokenizer or chat-template change invalidate a fine-tuning comparison?

The change alters the actual token sequence and potentially the target boundaries, even if the human-readable messages look identical. Special tokens, role delimiters and duplicated BOS markers can affect both learned behavior and measured loss.

I would freeze template and tokenizer revisions when comparing checkpoints. If a template migration is intentional, treat it as a separate experimental factor and inspect exact tokenized fixtures before attributing gains to the weight update.

A useful regression test stores input IDs, supervised-target indices and decoded boundary markers for several representative conversations. This establishes formatting compatibility without claiming that a few fixtures validate every possible message or tokenizer edge case.

44. How would you compute perplexity over overlapping evaluation windows?

I would score each target token once while using overlap only as additional context. Summing losses from every window without excluding repeated targets overweights tokens in the overlap and changes the metric.

For a long document, each window can supply a context prefix whose targets are ignored and a new suffix whose targets are counted. The total negative log-likelihood divided by the total counted tokens gives the mean used before exponentiation.

I would state context length, stride, boundary treatment and tokenizer. A document’s first tokens have less context than later tokens, and different windowing policies can change results. The purpose is a comparable language-model evaluation, not a universally context-independent score.

45. How can truncation silently change a supervised training objective?

Truncation can remove the completion, its ending marker or the evidence required to produce it. A syntactically valid token batch may then teach incomplete answers or score targets that no longer have sufficient context.

I would measure retained prompt and response lengths separately and inspect examples at the limit. For an extraction task, dropping the input field while retaining its answer turns the example into a guessing task rather than the intended evidence-based transformation.

Possible policies include rejecting oversize examples, shortening only designated context, or constructing explicit windows with valid labels. The choice depends on the task. Record how many examples and supervised tokens each policy removes, since filtering can bias the training distribution.

46. What does deduplication change beyond reducing dataset size?

Deduplication changes sampling weights and can reduce repeated exposure to the same material. It can also reduce train/evaluation contamination when duplicate or near-duplicate documents cross splits. Exact hashing catches only identical representations.

I would define normalization and near-duplicate thresholds before applying them. Aggressive filtering can remove legitimate repeated structures such as common code patterns or minority-language templates, while weak filtering leaves lightly edited copies.

Keep document lineage and evaluate the resulting mixture rather than assuming fewer rows means better data. Deduplication does not establish licensing, factual quality or privacy. Those are separate properties that require their own provenance and checks.

47. How do data-mixture weights translate into actual training exposure?

A mixture weight defines a sampling policy, but its meaning depends on whether sampling units are documents, examples or tokens. Equal document probabilities do not give equal token exposure when average document lengths differ.

I would track realized token counts by source and distinguish unique tokens from repeated epochs over a small source. Increasing a specialized corpus weight can improve domain fit while overexposing a narrow set of examples.

Compare mixtures under a fixed total compute or token budget and inspect both target and general-domain validation. A schedule that changes weights over time must be recorded with its progress unit so resumed training preserves the intended curriculum.

48. How do Adam moments affect a resumed training run?

Adam maintains moving estimates of gradient and squared-gradient statistics. Resuming only weights discards those estimates, changing update magnitudes and directions relative to a continuation with restored optimizer state.

The step counter matters for bias correction, and parameter-to-state mapping must match after model changes. Loading states into a differently ordered or resized parameter set can fail or attach history to the wrong tensors if handled carelessly.

I would compare the next update before and after a checkpoint round trip on a small deterministic fixture. Matching initial logits only verifies weights; it does not verify continuation of optimization. If optimizer history is intentionally reset, report that as a new training phase.

49. How does decoupled weight decay differ from adding an L2 term to an adaptive optimizer loss?

Decoupled weight decay applies a direct shrinkage update separately from the gradient-based adaptive update. Adding an L2 penalty contributes a parameter-proportional gradient that is then transformed by the optimizer’s adaptive statistics, so the two are not generally equivalent.

For Adam-like optimizers, this distinction affects the effective regularization of coordinates with different gradient histories. It also makes the parameter-group policy important: embeddings, normalization scales and biases may be treated differently by a particular training recipe.

I would preserve the optimizer variant and decay groups when reproducing a run. A parameter named “weight” is not sufficient evidence that it should receive decay, and changing the rule while keeping the same numeric coefficient is still a meaningful experimental change.

50. When is gradient accumulation equivalent to a larger batch?

It can reproduce the gradient of a larger batch when the objective is additive, normalization is correct, parameters stay fixed across microbatches, and stochastic or batch-dependent behavior is handled consistently. It is not automatically equivalent simply because example counts match.

For variable-length language modeling, accumulate summed eligible-token losses and normalize by the intended total token count. Dividing each microbatch mean by the number of accumulation steps weights microbatches equally instead of tokens when their sizes differ.

Floating-point reduction order and dropout draws can still differ. I would compare gradients on a small controlled example with stochastic layers disabled, then document any remaining numerical or algorithmic differences rather than promise bitwise equivalence.

51. What does activation checkpointing trade for memory?

Activation checkpointing saves selected boundaries and recomputes intermediate forward operations during backward instead of storing all intermediate activations. It trades additional computation for lower saved-activation memory.

The recomputed function must be consistent with the original forward pass. Randomness, mutable state and device movement can complicate that contract. Framework variants differ in supported autograd behavior and random-state handling.

I would test gradient agreement with checkpointing disabled on a small model and measure both peak memory and step time. Checkpointing does not reduce parameter or optimizer-state size, so it may not solve an out-of-memory problem dominated by those categories.

52. How would you locate the first source of nonfinite values in training?

I would find the earliest nonfinite activation, loss or gradient rather than inspect only the final optimizer failure. Reproduce the affected batch with recorded state, then instrument progressively smaller regions and compare precision settings.

Check input ranges, all-masked attention rows, invalid logarithms, extreme logits, loss normalization and overflow in reductions. A NaN after softmax may originate in its inputs or mask rather than in the optimizer.

I would preserve the failing example and step context before changing several settings at once. Gradient clipping cannot repair NaNs. A higher-precision diagnostic run can identify numerical sensitivity, but it is not proof that the original pipeline is otherwise correct.

53. How does data-parallel gradient synchronization work?

Each replica processes different examples with the same parameters, computes local gradients and combines them through collectives. The optimizer then uses the synchronized result so replicas remain consistent under the same update rule.

The reduction’s averaging convention must match loss normalization. Averaging equally sized local means is appropriate for equal token counts under a token objective, but unequal counts require weighting. Merely seeing matching replica parameters does not prove the intended objective was optimized.

I would test a distributed step against a single-process reference over the same examples and inspect collective ordering. Conditional computation or uneven inputs can cause hangs if ranks enter different synchronization paths.

54. What is the difference between sharding optimizer states, gradients and parameters?

These strategies remove different forms of replication across workers. Optimizer-state sharding distributes update history; gradient sharding distributes gradient storage; parameter sharding additionally distributes the model weights themselves, gathering them when needed.

Sharded category

What is distributed

Important remaining cost

Optimizer state

Moment/history tensors

Weights and gradients may remain replicated

Gradients

Gradient partitions

Collective reductions and update coordination

Parameters

Weight partitions

Temporary gathers, activations and communication

Each additional category can reduce persistent local memory but introduces communication and scheduling requirements. Peak memory still includes temporarily gathered parameters, activations and buffers, so dividing total parameter bytes by worker count is incomplete.

I would choose sharding granularity using measured memory and interconnect costs. A model that fits only with aggressive sharding may spend more time gathering than computing if units are too small or overlap is ineffective.

55. How should a learning-rate schedule behave when an update is skipped?

The schedule should advance according to its explicitly defined progress unit. If it is defined in successful optimizer updates, an overflow-skipped step should not consume a learning-rate step as if weights changed.

A token-based schedule may intentionally track consumed data even when an update is skipped, but that is a different policy. The key is to align scheduler calls, scaler behavior and logged step counts rather than let library defaults create an accidental mixture.

I would test a forced overflow or simulated skipped update and inspect the next learning rate. Save the scheduler and scaler state together, and make dashboards distinguish attempted steps from applied updates.

56. Why is one LoRA matrix often initialized to zero?

For an update written B A, initializing B to zero and A to a suitable random matrix makes the initial adapter contribution zero. The adapted model therefore begins with the base model’s function while allowing gradients to start learning the update.

At that first step, B can receive a nonzero gradient through A, while the gradient to A through B is initially zero. Initializing both matrices to zero would block this simple factorized update from learning because each factor’s gradient depends on the other.

# Scalar analogue of update B @ A with loss (B*A - target)**2.
a, b, target = 0.5, 0.0, 1.0
error = b*a - target
grad_a, grad_b = 2*error*b, 2*error*a
assert grad_a == 0.0 and grad_b == -1.0
# Both factors zero would make both initial gradients zero.
print(grad_a, grad_b)  # -0.0 -1.0
# Scalar analogue of update B @ A with loss (B*A - target)**2.
a, b, target = 0.5, 0.0, 1.0
error = b*a - target
grad_a, grad_b = 2*error*b, 2*error*a
assert grad_a == 0.0 and grad_b == -1.0
# Both factors zero would make both initial gradients zero.
print(grad_a, grad_b)  # -0.0 -1.0
# Scalar analogue of update B @ A with loss (B*A - target)**2.
a, b, target = 0.5, 0.0, 1.0
error = b*a - target
grad_a, grad_b = 2*error*b, 2*error*a
assert grad_a == 0.0 and grad_b == -1.0
# Both factors zero would make both initial gradients zero.
print(grad_a, grad_b)  # -0.0 -1.0

Initialization variants exist, so I would check the chosen method rather than assume every adapter uses this recipe. A test should verify both initial output parity and the expected nonzero gradients after a backward pass.

57. How would you choose LoRA target modules without treating rank as the only capacity control?

Target-module selection determines where adaptation can act. Attention projections, feed-forward projections and output layers affect different parts of the computation. Rank controls each selected update’s factorization, but module coverage also changes parameter count and expressive capacity.

I would compare a small number of justified configurations under the same data and evaluation protocol. A narrow attention-only adapter and a broader lower-rank adapter can have similar parameter budgets but different behavior.

Inspect trainable parameter names and gradient flow before running a large experiment. A configuration pattern that matches no modules can appear to load while failing to train the intended adapter. Preserve selected modules, rank, scaling and base-model revision in the artifact.

58. Why can merging an adapter before quantization differ from merging after quantization?

Quantization is nonlinear: rounding the base-plus-update generally differs from rounding the base first and then adding the update. Dequantization also reconstructs an approximation of the original weights, not the exact original tensor.

I would define the deployment path explicitly and compare its logits and task behavior against the intended adapted reference. A successful file conversion does not establish equivalence, particularly when update magnitudes are small relative to quantization steps.

Keep the original base and adapter artifacts so alternative export paths can be reproduced. When the runtime supports separate adapters on a quantized base, that can avoid one merge step, but it introduces its own supported-kernel and numerical constraints.

59. How is a completion log-probability calculated for preference training?

For a causal model, sum the log-probabilities assigned to the observed completion tokens conditioned on the prompt and earlier completion tokens. Prompt-only positions are excluded when the objective is defined over the completion.

The label shift and mask must align with the actual token sequence. Padding is excluded, and the treatment of ending tokens must be consistent between chosen and rejected responses. Length normalization changes the quantity and should not be introduced silently.

I would validate the calculation on a tiny vocabulary and short response before using it in DPO or another objective. Incorrect masks can reward prompt prediction or padding rather than the preference distinction the dataset is meant to teach.

60. What does the reference-policy ratio contribute to the basic DPO objective?

The basic DPO objective compares the policy’s relative log-likelihood of chosen and rejected completions against that same relative likelihood under a reference policy. It therefore measures a preference-oriented change from the reference, not just whether the chosen response has a larger raw probability.

If both policy and reference favor the chosen answer by the same log-probability margin, their difference is zero and the basic logistic term is at its neutral point. The beta coefficient scales this difference in the loss.

from math import log, exp, isclose
policy_chosen, policy_rejected = -2.0, -3.0
ref_chosen, ref_rejected = -4.0, -5.0
beta = 0.1
margin = (policy_chosen - policy_rejected) - (ref_chosen - ref_rejected)
loss = log(1 + exp(-beta * margin))  # small illustrative scores
assert margin == 0.0 and isclose(loss, log(2))
print(round(loss, 6))  # 0.693147
from math import log, exp, isclose
policy_chosen, policy_rejected = -2.0, -3.0
ref_chosen, ref_rejected = -4.0, -5.0
beta = 0.1
margin = (policy_chosen - policy_rejected) - (ref_chosen - ref_rejected)
loss = log(1 + exp(-beta * margin))  # small illustrative scores
assert margin == 0.0 and isclose(loss, log(2))
print(round(loss, 6))  # 0.693147
from math import log, exp, isclose
policy_chosen, policy_rejected = -2.0, -3.0
ref_chosen, ref_rejected = -4.0, -5.0
beta = 0.1
margin = (policy_chosen - policy_rejected) - (ref_chosen - ref_rejected)
loss = log(1 + exp(-beta * margin))  # small illustrative scores
assert margin == 0.0 and isclose(loss, log(2))
print(round(loss, 6))  # 0.693147

I would keep tokenization, completion masks and reference weights fixed for a controlled run. A falling preference loss can still reflect shortcuts in the comparison data; it does not independently establish improved factual accuracy or preserved general capabilities.

61. How would you test a KV cache against a full-sequence reference?

I would compute reference logits for a fixed complete sequence, then process the same sequence incrementally with the cache and compare corresponding logits. Use evaluation mode, consistent positions and a stated floating-point tolerance.

Test a one-token prefix, several prompt lengths, multiple decode steps and unequal batch lengths. A bug can appear only after cache growth or a boundary crossing, so one successful first token is insufficient.

Compare pre-selection logits before comparing sampled text. Sampling can amplify tiny differences into unrelated continuations. This test checks functional cache parity for the exercised cases; it does not measure throughput or prove every supported architecture uses the same cache layout.

62. Why can a square causal mask be wrong for cached multi-token decoding?

With cached keys, query length and key length differ. A mask must allow each new query to attend to its valid prefix plus earlier new tokens. A square lower-triangular mask based only on the new query count omits the cached-prefix offset.

For a cached prefix of four tokens and two new queries, the first query can attend to five total positions and the second to six. A 2 × 6 mask therefore needs its diagonal aligned with absolute sequence positions.

prefix, new_tokens = 4, 2
allowed = [[key <= prefix + query
            for key in range(prefix + new_tokens)]
           for query in range(new_tokens)]
assert allowed == [[True, True, True, True, True, False],
                   [True, True, True, True, True, True]]
print(allowed)
prefix, new_tokens = 4, 2
allowed = [[key <= prefix + query
            for key in range(prefix + new_tokens)]
           for query in range(new_tokens)]
assert allowed == [[True, True, True, True, True, False],
                   [True, True, True, True, True, True]]
print(allowed)
prefix, new_tokens = 4, 2
allowed = [[key <= prefix + query
            for key in range(prefix + new_tokens)]
           for query in range(new_tokens)]
assert allowed == [[True, True, True, True, True, False],
                   [True, True, True, True, True, True]]
print(allowed)

I would test explicit masks against the attention API’s non-square causal convention. Different backends may define alignment differently; assuming that a boolean “causal” flag always means the intended cached alignment can silently corrupt generation.

63. What happens to a KV cache when beam search reorders hypotheses?

The cache must follow the selected parent hypothesis for each new beam. Reordering token histories without applying the corresponding cache permutation pairs a query with another hypothesis’s keys and values.

When two beams share a parent, they initially share identical prefix state but can diverge after appending different tokens. An implementation may copy that state or use safe shared storage with separate writes.

I would use a tiny deterministic beam example and compare cached results with recomputation from each beam’s explicit history. This exposes ordering and aliasing bugs without relying on whether the final generated sentence happens to sound reasonable.

64. How does sliding-window attention change cache retention?

A sliding-window layer only needs keys and values within its supported attention window, rather than all earlier positions. A hybrid model may combine such layers with full-attention layers, so retention requirements can differ by layer.

Evicting old cache entries does not mean position numbering restarts. The retained tokens still need the positional treatment expected by the architecture. Ring-buffer storage indices and absolute model positions are separate quantities.

I would test the exact transition at the window boundary and compare against a reference with the same window mask. Dropping old state from a model trained for full attention is a behavioral approximation, not automatically an equivalent memory optimization.

65. Why does FlashAttention avoid storing the full attention matrix?

FlashAttention tiles the computation and maintains softmax-related statistics while moving data through faster on-chip memory. It computes attention without materializing the entire sequence-by-sequence probability matrix in high-bandwidth memory.

The key improvement concerns memory traffic and intermediate storage. Dense attention still involves pairwise query-key interactions; the method does not turn conventional dense attention into a linear-time approximation.

I would compare numerical outputs and gradients within tolerance and benchmark the actual shapes and hardware. Kernel availability, head dimension and masks affect which path executes. Merely enabling a configuration flag is not evidence that the intended fast kernel was selected.

66. How does an online softmax combine two blocks of scores?

Keep a running maximum m, a scaled exponential sum l and a similarly scaled weighted-value sum. When a new block has a larger maximum, rescale the old sums by exp(old_m - new_m) before adding the new block’s contributions.

This puts both blocks on the same exponential scale without storing every score. Dividing the final weighted-value sum by the final l yields the attention result for that row.

from math import exp, isclose
blocks = [([1000.0, 999.0], [2.0, 4.0]),
          ([1001.0], [8.0])]
m, denom, numerator = float('-inf'), 0.0, 0.0
for scores, values in blocks:
    new_m = max(m, max(scores))
    rescale = exp(m - new_m)
    weights = [exp(s - new_m) for s in scores]
    denom = denom * rescale + sum(weights)
    numerator = numerator * rescale + sum(w*v for w, v in zip(weights, values))
    m = new_m
scores, values = [1000.0, 999.0, 1001.0], [2.0, 4.0, 8.0]
weights = [exp(s - max(scores)) for s in scores]
reference = sum(w*v for w, v in zip(weights, values)) / sum(weights)
assert isclose(numerator / denom, reference, rel_tol=1e-12)
print(round(reference, 6))  # 6.171507
from math import exp, isclose
blocks = [([1000.0, 999.0], [2.0, 4.0]),
          ([1001.0], [8.0])]
m, denom, numerator = float('-inf'), 0.0, 0.0
for scores, values in blocks:
    new_m = max(m, max(scores))
    rescale = exp(m - new_m)
    weights = [exp(s - new_m) for s in scores]
    denom = denom * rescale + sum(weights)
    numerator = numerator * rescale + sum(w*v for w, v in zip(weights, values))
    m = new_m
scores, values = [1000.0, 999.0, 1001.0], [2.0, 4.0, 8.0]
weights = [exp(s - max(scores)) for s in scores]
reference = sum(w*v for w, v in zip(weights, values)) / sum(weights)
assert isclose(numerator / denom, reference, rel_tol=1e-12)
print(round(reference, 6))  # 6.171507
from math import exp, isclose
blocks = [([1000.0, 999.0], [2.0, 4.0]),
          ([1001.0], [8.0])]
m, denom, numerator = float('-inf'), 0.0, 0.0
for scores, values in blocks:
    new_m = max(m, max(scores))
    rescale = exp(m - new_m)
    weights = [exp(s - new_m) for s in scores]
    denom = denom * rescale + sum(weights)
    numerator = numerator * rescale + sum(w*v for w, v in zip(weights, values))
    m = new_m
scores, values = [1000.0, 999.0, 1001.0], [2.0, 4.0, 8.0]
weights = [exp(s - max(scores)) for s in scores]
reference = sum(w*v for w, v in zip(weights, values)) / sum(weights)
assert isclose(numerator / denom, reference, rel_tol=1e-12)
print(round(reference, 6))  # 6.171507

I would test the recurrence against ordinary stable softmax using scores with very different magnitudes. Forgetting to rescale the old numerator as well as the denominator can produce plausible but incorrect outputs. Floating-point ordering can differ while the mathematical operation remains equivalent.

67. What does a paged KV-cache block table represent?

It maps a sequence’s logical token blocks to physical cache blocks. The model can view a contiguous logical history even when its stored keys and values occupy noncontiguous memory.

This reduces the need to reserve one large contiguous allocation per request and supports flexible sharing and reclamation. It does not eliminate the underlying bytes needed for valid tokens or every source of fragmentation.

I would test block-boundary reads, partial final blocks and reclamation after cancellation. A stale block-table entry can point to another request’s data, making correctness and isolation tests as important as allocator efficiency.

68. Why does sharing a prefix require copy-on-write behavior when sequences branch?

Branches can share immutable cache blocks for their common prefix. Once a branch appends into a partially filled shared block, modifying that block in place could change another branch’s state. Copy-on-write preserves each logical history.

A block-aligned complete prefix may remain shared while each branch receives a separate new block. A partial block requires more care because its unused positions may be filled differently by different continuations.

I would test two branches that append distinct tokens, then compare each against independent recomputation. Reference counts, ownership and reclamation must remain consistent when one branch ends early; otherwise memory savings can become corruption or use-after-free.

69. How does chunked prefill change interference with active decoding?

Chunked prefill divides a long prompt’s processing into smaller scheduling units so the server can interleave them with decode work. This can reduce the time an active stream waits behind one large prompt computation.

The chunk size creates a trade-off: smaller chunks increase scheduling flexibility but may reduce kernel efficiency or add overhead. The prompt still requires its necessary computation; chunking does not make its tokens free.

I would measure inter-token latency and prompt completion time under a mixed workload, not only aggregate throughput. Also test positional and cache continuity across chunk boundaries against an unchunked reference, since scheduling must preserve the intended model computation.

70. Why can a static cache help compilation while wasting memory?

A fixed-capacity cache can keep tensor shapes stable across decode steps, making graph specialization and memory planning easier. The reserved capacity may exceed the valid sequence length, so unused positions consume memory and may require masked computation.

A service with similar request lengths may benefit more than one mixing very short and very long contexts. Bucketing can reduce waste but adds configuration and routing complexity.

I would benchmark warm and cold paths separately, record recompilations and include the cache reservation in concurrency estimates. Compilation success is not proof of a latency improvement; capture overhead and padding to fixed shapes can outweigh saved launch or dispatch costs.

71. What is the difference between weight-only and activation quantization?

Weight-only quantization reduces stored weight precision while activations may remain in a higher-precision format. Activation quantization also represents intermediate values at reduced precision and needs a policy for their changing ranges.

The two paths have different kernels, metadata and error sources. A four-bit checkpoint does not imply four-bit accumulations or four-bit KV storage. Dequantization and scale handling can dominate some small workloads.

I would document which tensors are quantized, how scales are selected and the accumulator dtype. Compare both numerical behavior and measured performance; file-size reduction alone does not establish faster execution or an acceptable error profile.

72. How do scale and zero point define affine quantization?

Affine quantization approximates a real value with an integer q and parameters scale s and zero point z, commonly reconstructing x as s × (q - z). Encoding rounds x/s + z and clamps it to the representable integer range.

The zero point allows real zero to be represented even when the quantized range is asymmetric. Rounding and clipping introduce different errors: values near bin boundaries are rounded, while out-of-range values saturate.

scale, zero_point = 0.5, 4
qmin, qmax = 0, 7
def encode(x):
    return min(qmax, max(qmin, round(x / scale) + zero_point))
def decode(q):
    return scale * (q - zero_point)
assert decode(encode(0.0)) == 0.0
assert decode(encode(10.0)) == 1.5  # saturation at qmax
print(encode(0.0), decode(encode(10.0)))  # 4 1.5
scale, zero_point = 0.5, 4
qmin, qmax = 0, 7
def encode(x):
    return min(qmax, max(qmin, round(x / scale) + zero_point))
def decode(q):
    return scale * (q - zero_point)
assert decode(encode(0.0)) == 0.0
assert decode(encode(10.0)) == 1.5  # saturation at qmax
print(encode(0.0), decode(encode(10.0)))  # 4 1.5
scale, zero_point = 0.5, 4
qmin, qmax = 0, 7
def encode(x):
    return min(qmax, max(qmin, round(x / scale) + zero_point))
def decode(q):
    return scale * (q - zero_point)
assert decode(encode(0.0)) == 0.0
assert decode(encode(10.0)) == 1.5  # saturation at qmax
print(encode(0.0), decode(encode(10.0)))  # 4 1.5

I would test zero, endpoints, halfway cases and outliers. The exact rounding convention and per-tensor or per-channel scale layout belong to the artifact contract; using the correct integer bytes with the wrong scales produces the wrong model.

73. Why can per-channel quantization preserve more useful detail than one global scale?

Different channels can have different value ranges. A global scale determined by a large outlier may leave only a few effective levels for smaller channels. Per-channel scales adapt the resolution to each channel’s range.

The benefit comes with additional metadata and kernel-layout requirements. Group-wise methods make another trade-off between finer scales and overhead; smaller groups are not automatically faster or universally more accurate.

I would inspect reconstruction error by channel and evaluate the resulting model, especially channels with outliers. A low average weight error is only a diagnostic because activation distributions and downstream sensitivity determine how that error affects predictions.

74. How would you detect calibration data that is unsuitable for activation quantization?

I would compare calibration activation ranges with representative held-out deployment inputs and examine saturation or clipping rates. Calibration built from short ordinary prose may miss ranges produced by long code or structured documents.

Use task and length slices, and preserve a separate evaluation set so selecting quantization parameters does not tune directly to the final test. Inspect difficult layers rather than rely only on one global average.

A calibration set cannot guarantee coverage of every future input. The deployment should retain the quantization recipe and a reference configuration for regression analysis. If failures concentrate in a known slice, revise calibration or restrict the optimized path based on measured evidence.

75. Why must speculative sampling use both draft and target probabilities?

Exact speculative sampling corrects the draft proposal distribution toward the target distribution. The acceptance rule depends on their probability ratio, and rejection requires a residual correction distribution. Looking only at the target’s favorite token does not implement the same sampler.

For a proposed token with target probability p and draft probability q, the basic acceptance probability is min(1, p/q) where the proposal has q greater than zero. Rejected mass is redistributed using the positive part of target minus draft probabilities.

from math import isclose
p, q = [0.7, 0.3], [0.4, 0.6]
accepted_mass = [min(pi, qi) for pi, qi in zip(p, q)]
reject_mass = 1 - sum(accepted_mass)
residual = [max(0.0, pi - qi) for pi, qi in zip(p, q)]
residual_total = sum(residual)
corrected = [a + reject_mass * r / residual_total
             for a, r in zip(accepted_mass, residual)]
assert all(isclose(a, b) for a, b in zip(corrected, p))
print([round(x, 3) for x in corrected])  # [0.7, 0.3]
from math import isclose
p, q = [0.7, 0.3], [0.4, 0.6]
accepted_mass = [min(pi, qi) for pi, qi in zip(p, q)]
reject_mass = 1 - sum(accepted_mass)
residual = [max(0.0, pi - qi) for pi, qi in zip(p, q)]
residual_total = sum(residual)
corrected = [a + reject_mass * r / residual_total
             for a, r in zip(accepted_mass, residual)]
assert all(isclose(a, b) for a, b in zip(corrected, p))
print([round(x, 3) for x in corrected])  # [0.7, 0.3]
from math import isclose
p, q = [0.7, 0.3], [0.4, 0.6]
accepted_mass = [min(pi, qi) for pi, qi in zip(p, q)]
reject_mass = 1 - sum(accepted_mass)
residual = [max(0.0, pi - qi) for pi, qi in zip(p, q)]
residual_total = sum(residual)
corrected = [a + reject_mass * r / residual_total
             for a, r in zip(accepted_mass, residual)]
assert all(isclose(a, b) for a, b in zip(corrected, p))
print([round(x, 3) for x in corrected])  # [0.7, 0.3]

I would test the resulting frequencies on a tiny vocabulary against the target distribution. This verifies the sampling logic in a bounded setting; it does not establish a speedup, which also depends on draft cost, acceptance and verification efficiency.

76. How should a speculative decoder handle a rejected draft suffix?

After the first rejected draft token, later proposed tokens are conditioned on a history that was not accepted. Their cached state cannot simply be retained as though it belonged to the corrected sequence.

The decoder must commit only the accepted prefix, append the correction under the algorithm’s rules and roll back or invalidate unused draft/verification state. Cache positions and lengths must reflect the committed sequence.

I would force rejections at the first, middle and last draft positions and compare subsequent logits against ordinary decoding. This targets state-management correctness, which a high average acceptance rate can otherwise hide.

77. How would you benchmark a GPU kernel without timing only its launch?

GPU work is often asynchronous with respect to the host. A host timer around a launch can measure enqueue time rather than execution unless synchronization or correctly placed device events establish the measurement boundary.

I would warm up the kernel, use repeated trials, control relevant shapes and dtypes, and report the timing method. Compilation, allocation and transfers should be included or excluded deliberately according to the question.

A microbenchmark is useful for a component comparison, but end-to-end latency also includes scheduling and dependencies. Do not infer a service-level improvement solely from a faster isolated kernel, especially if its memory use reduces batch concurrency.

78. What is arithmetic intensity, and why does it matter for decoding?

Arithmetic intensity is the amount of computation performed per byte transferred at a specified memory level. Comparing it with hardware compute and bandwidth capabilities helps identify whether an operation is likely limited by data movement or arithmetic.

At small decode batches, repeatedly reading large weight matrices for relatively little work per request can make bandwidth important. Larger batches can reuse weights across more tokens, increasing useful computation per weight read.

This is a model for forming hypotheses, not a substitute for profiling. Cache traffic, quantization, communication and kernel efficiency can change the bottleneck. Specify which bytes and FLOPs are counted before presenting a roofline estimate.

79. Why can vocabulary projection and sampling become significant after optimizing attention?

The output layer computes scores across the vocabulary, and selection or sampling processes those scores. Once other operations are faster, this projection and its memory traffic can become a larger fraction of decode time.

A larger vocabulary increases the output dimension even if sequence length and hidden width stay constant. During training, materializing logits for every position can also consume substantial memory.

I would profile the actual output path before attempting approximate shortcuts. Fused loss or sampling kernels may reduce intermediate storage, while restricting the vocabulary changes semantics unless it is an intended constraint. Optimizing one bottleneck often exposes another rather than removing total system cost.

80. What evidence should accompany a claim that an inference optimization preserves behavior?

I would specify the reference, supported configurations, numerical tolerance and tested invariants. Compare logits on representative and boundary cases, then test generation and task outcomes where small numerical changes can matter.

Include masks, padding, cache growth, long contexts, quantization and any adapter combinations the optimization claims to support. A single greedy output match is weaker evidence than aligned logits and targeted state-transition tests.

Behavioral preservation can mean mathematical equivalence within rounding, distributional equivalence, or merely acceptable task quality. Those claims are different. The report should state which one is supported and avoid turning a passing local fixture into a guarantee for every model or accelerator.

Large Language Models Interview Questions and Answers for Experienced

Experienced answers examine the limits of evidence as well as advanced methods. They distinguish algorithmic guarantees, implementation assumptions and empirical observations before recommending an operational decision.

81. How would you derive token-correct gradients when distributed ranks have unequal target counts?

I would start from the global objective: total eligible-token loss divided by total eligible-token count. If the distributed wrapper averages gradients over W ranks, each rank can backpropagate W times its local summed loss divided by the globally summed count, under that averaging convention.

For ranks with two and eight valid targets, averaging their local mean losses weights the small rank too heavily. Reducing the counts establishes the correct denominator; the loss numerator remains local for backward and is scaled to compensate for gradient averaging.

from math import isclose
# Local summed-loss gradients at the same parameter value.
local_gradient_sums = [4.0, 24.0]
valid_counts = [2, 8]
world_size = len(valid_counts)
global_count = sum(valid_counts)
scaled = [world_size * g / global_count for g in local_gradient_sums]
after_ddp_average = sum(scaled) / world_size
reference = sum(local_gradient_sums) / global_count
assert isclose(after_ddp_average, reference)
wrong_equal_rank_mean = sum(g/n for g, n in zip(local_gradient_sums, valid_counts)) / world_size
assert isclose(reference, 2.8) and isclose(wrong_equal_rank_mean, 2.5)
print(reference, wrong_equal_rank_mean)  # 2.8 2.5
from math import isclose
# Local summed-loss gradients at the same parameter value.
local_gradient_sums = [4.0, 24.0]
valid_counts = [2, 8]
world_size = len(valid_counts)
global_count = sum(valid_counts)
scaled = [world_size * g / global_count for g in local_gradient_sums]
after_ddp_average = sum(scaled) / world_size
reference = sum(local_gradient_sums) / global_count
assert isclose(after_ddp_average, reference)
wrong_equal_rank_mean = sum(g/n for g, n in zip(local_gradient_sums, valid_counts)) / world_size
assert isclose(reference, 2.8) and isclose(wrong_equal_rank_mean, 2.5)
print(reference, wrong_equal_rank_mean)  # 2.8 2.5
from math import isclose
# Local summed-loss gradients at the same parameter value.
local_gradient_sums = [4.0, 24.0]
valid_counts = [2, 8]
world_size = len(valid_counts)
global_count = sum(valid_counts)
scaled = [world_size * g / global_count for g in local_gradient_sums]
after_ddp_average = sum(scaled) / world_size
reference = sum(local_gradient_sums) / global_count
assert isclose(after_ddp_average, reference)
wrong_equal_rank_mean = sum(g/n for g, n in zip(local_gradient_sums, valid_counts)) / world_size
assert isclose(reference, 2.8) and isclose(wrong_equal_rank_mean, 2.5)
print(reference, wrong_equal_rank_mean)  # 2.8 2.5

Handle zero-target ranks while preserving required collectives, and test against one process evaluating the combined batch. Accumulation adds another boundary: the denominator must cover the intended whole update, not just whichever microbatch happens to be processed first.

82. How can a distributed training job hang even when every GPU has enough memory?

Ranks may execute incompatible collective sequences, counts or tensor types. One rank can wait in a gradient reduction while another has skipped that path, entered a different collective or failed earlier in data loading.

I would inspect rank-specific progress and the first divergence, not assume the last visible collective caused the problem. Conditional branches, uneven input exhaustion and errors swallowed on one worker are common places to investigate.

Use a small reproducible topology and explicit timeouts for diagnosis. Increasing communication timeouts only hides a permanent mismatch. A repair should include a test of the triggering uneven or exceptional path, because a normal fixed-length batch may never exercise it.

83. What makes exact training resume difficult when the number of workers changes?

Changing worker count changes data partitioning, reduction order and potentially optimizer-state layout. Restoring parameter values alone does not reconstruct which examples each worker would have consumed or the exact floating-point update sequence.

I would distinguish semantic continuation from bitwise replay. Resharded checkpoints can preserve the logical model and optimizer state while a new sampler schedule produces a valid but different trajectory. Global sample or token position is more portable than a worker-local iterator offset.

Record the topology change and verify no unintended data omission or duplication. If exact replay is a requirement, constrain the environment accordingly; do not promise it merely because a checkpoint loader supports a new world size.

84. How would you prove a checkpoint is a consistent snapshot during asynchronous saving?

I would require all saved components to correspond to one defined update boundary. If weights, optimizer moments and counters are copied at different times while training continues, the resulting files can each be readable but collectively inconsistent.

A design can stage an immutable snapshot or use a framework’s supported asynchronous checkpoint contract, then publish a manifest only after all required shards are durable. Partial output should not be advertised as the latest complete checkpoint.

Test interruption during staging, writing and manifest publication. Recovery should choose the last complete compatible snapshot and validate shard identities and checksums. This is a consistency problem, not simply a question of whether individual files exist.

85. How do pipeline schedules trade activation memory against idle time?

A pipeline partitions layers into stages and schedules microbatches through forward and backward work. Schedules differ in how many activations remain live and how much time stages spend waiting during startup, drain or imbalance.

Increasing microbatch count can reduce the relative bubble under a chosen schedule, but may change batch semantics or increase overhead. A stage with an expensive embedding or output layer can dominate despite equal layer counts.

I would model per-stage compute, transfers and activation residency, then validate with traces. Compare schedules at the same optimization objective and global batch rather than accepting a throughput gain that silently changes training behavior.

86. Why can tensor-parallel performance collapse across a slow interconnect?

Tensor-parallel layers require communication between shards to assemble or reduce intermediate results. These collectives occur frequently enough that latency and bandwidth can dominate the saved local compute, especially for small per-device work.

I would inspect message sizes and placement relative to the actual topology. A configuration that is effective within a high-bandwidth node may perform poorly when the same group spans servers.

The solution may involve changing parallelism groups, combining tensor and pipeline parallelism differently, or using more independent replicas once the model fits. Device count alone is not a scaling argument; benchmark the required communication pattern at realistic batch and sequence sizes.

87. What does sequence parallelism shard that ordinary tensor parallelism may leave replicated?

Sequence parallelism can distribute certain activations and associated operations across the sequence dimension while cooperating with tensor parallelism. Its purpose is to reduce replicated activation storage and work in parts of the transformer.

The precise term depends on the framework; it should not be conflated with every method that splits a long context. Context-parallel attention can require additional exchange of keys and values, whereas a particular sequence-parallel scheme may target normalization and dropout around tensor-parallel layers.

I would draw the activation layout and collectives for the chosen implementation. Naming a parallelism mode is insufficient to establish memory savings or correctness; test how masks, gradients and gathers reconstruct the reference computation.

88. How would you diagnose a mixture-of-experts model whose experts receive highly uneven traffic?

I would inspect routing probabilities, actual assignments and per-expert token counts by layer and data slice. Imbalance can create overloaded experts, idle devices and altered training behavior if capacity limits drop or reroute tokens.

The remedy depends on whether the cause is data skew, router optimization, capacity policy or implementation error. Auxiliary balancing objectives and routing constraints can help, but they also change the optimization problem and may trade specialization for uniformity.

Compare quality and throughput together, including rare-token slices. Uniform routing is not itself the goal; useful specialization can be uneven. The requirement is a deliberate balance policy whose operational and learning consequences are measured.

89. Why do active parameters not fully predict the cost of a mixture-of-experts model?

Active parameter count describes the subset used for a token, but the deployment may still store many experts and move tokens between devices. Routing, all-to-all communication, load imbalance and small expert batches affect realized performance.

Two models with similar active counts can require different memory footprints and interconnect capacity. At low batch sizes, an expert may receive too little work for an efficient matrix multiplication even when total theoretical arithmetic is modest.

I would profile dispatch, expert computation and combine phases separately. A comparison should include total resident weights, routing distribution and topology, rather than treating a sparse model as a dense model with the same active parameter number.

90. How would you test that an all-to-all expert dispatcher preserves token identity?

I would attach deterministic token and source-position identifiers to a small synthetic batch, route it through the dispatch and combine path, and verify that each expert output returns to the correct original position with the correct gate weight.

Include repeated expert assignments, empty experts, capacity overflow and unequal token counts. A permutation error may preserve aggregate shapes and even plausible numerical ranges while mixing information between tokens.

Compare the distributed path against a simple local reference with fixed routing. This isolates transport and indexing from router learning. Only after that passes would I interpret training-loss changes as evidence about the model rather than the dispatcher.

91. Why can model FLOPs utilization be misleading when comparing training systems?

The metric depends on how useful model FLOPs and hardware peak capability are defined. Different accounting for recomputation, sparsity, attention and precision can make two reported utilization numbers incomparable.

A system doing extra recomputation may achieve high hardware activity while processing fewer useful tokens per second. Conversely, a memory-saving change can lower a FLOP-based metric yet enable a larger useful batch.

I would report the formula, precision-specific peak assumption, sequence and batch shapes, step time and tokens per second. The decision metric should connect to training progress and cost, with utilization used to explain behavior rather than serve as a standalone ranking.

92. How would you design a controlled ablation of a new attention architecture?

I would isolate the architectural change while holding data, tokenizer, training budget and evaluation protocol as comparable as possible. Parameter count and compute may not both stay identical, so the comparison must state which resource is controlled.

Use multiple scales or seeds where feasible and inspect optimization stability as well as final quality. An advantage from a better-tuned learning rate should not be attributed entirely to the architecture.

Include implementation correctness tests and measured runtime. A theoretical reduction in operations can lose on existing kernels. The conclusion should distinguish representational quality, training efficiency and serving efficiency rather than collapse them into one “better model” claim.

93. What evidence is needed before extrapolating a scaling law to a larger run?

I would examine the fitted range, residuals, uncertainty and whether the proposed run changes data quality, architecture or optimization regime. A smooth fit over smaller experiments is a hypothesis about a larger regime, not a physical law.

Hold out some scales or budgets to test prediction, and compare plausible alternative functional forms. Data repetition and saturation can invalidate an extrapolation that assumes additional tokens behave like fresh samples from the same distribution.

Use the fit to inform a budget decision with uncertainty bounds and intermediate checkpoints. A pilot that deviates from the predicted curve should trigger investigation before committing the remaining compute.

94. How would you distinguish undertraining from an optimization failure?

I would compare training and validation trajectories, learning-rate behavior and controlled smaller runs. A model that continues to improve predictably with more tokens differs from one whose updates are ineffective because gradients, masks or numerical stability are broken.

Check whether loss responds to a tiny overfit test and whether parameter updates have plausible magnitude. Compare against a known-good implementation on a bounded dataset before interpreting a flat curve as an inherent capacity limit.

Undertraining is relative to a budget and target; optimization failure is a defect or unsuitable configuration in using that budget. Adding compute without diagnosis can make both expensive while resolving neither.

95. How should continued pretraining be evaluated for catastrophic forgetting?

I would retain evaluation sets for capabilities that must survive alongside the new-domain targets. Continued pretraining can improve domain likelihood while degrading language coverage or task behavior elsewhere.

Compare checkpoints over time, not only the final one, and inspect whether the degradation tracks data-mixture changes or excessive optimization. Rehearsal mixtures or other retention methods are hypotheses to test under the same budget.

A single aggregate benchmark can hide a severe regression in a small but important slice. Define acceptable trade-offs before selecting the checkpoint, and keep the original model available so a narrow domain gain is not mistaken for universal improvement.

96. How can a preference dataset accidentally reward response length instead of quality?

If preferred answers are systematically longer, the model can exploit length-correlated features rather than the intended quality criterion. The objective may improve while concise correct answers become less likely.

I would stratify preference outcomes by length difference and inspect matched-content pairs. Review whether rejected answers are merely shorter, contain formatting artifacts or come from a distinguishable generator.

Correct the data or objective only with a defined target behavior and independent evaluation. Automatically length-normalizing every likelihood changes the objective and can introduce a different bias; it is not a universal fix for poor comparison data.

97. How would you diagnose reward hacking in an online language-model training loop?

I would compare optimized reward with independent task checks and inspect outputs at increasing optimization strength. A growing gap can reveal exploitation of evaluator shortcuts, such as verbose repetition, format tricks or claims the evaluator cannot verify.

Preserve samples from earlier checkpoints and evaluate blindly with criteria not used directly as the reward. For verifiable tasks, execution or trusted labels can expose improvements that exist only in the learned reward model.

Possible responses include revising data, limiting policy movement or changing the reward mechanism, but each requires retesting. A higher reward is an optimization result, not proof that the underlying human objective improved.

98. What makes distillation from teacher logits different from training on teacher-generated text?

Logit-based distillation can transfer a distribution over alternatives, while training on generated text observes selected sequences as targets. The latter discards information about other tokens the teacher considered plausible.

Vocabulary and tokenizer compatibility affect how distributions can be compared. If tokenizations differ, aligning per-position logits is not a simple elementwise operation. Generated-text training is easier to decouple but inherits the teacher’s sampling and filtering choices.

I would compare student quality, data cost and coverage rather than assume one signal is always superior. The teacher is not ground truth; independent evaluation must detect inherited errors and capabilities lost under the student’s smaller capacity.

99. How would you distinguish benchmark contamination from legitimate generalization?

I would investigate overlap between evaluation material and training sources, including paraphrases and answer explanations, while recognizing that absence of an exact string match is weak evidence. Provenance and temporal separation help but do not make contamination impossible.

Use fresh or transformed evaluation items that preserve the intended skill and compare performance patterns. Unusual verbatim completions or sharp drops on equivalent variants can motivate investigation, but neither alone proves memorization.

The report should state what was searched and what remains unknown. Avoid treating a contamination detector as a definitive label; the practical goal is an evaluation design whose conclusions do not depend on undisclosed exposure.

100. How would you test long-context positional scaling independently of retrieval difficulty?

I would construct tasks that vary absolute position and distance while controlling the amount and form of required evidence. This helps separate positional failure from failures caused by noisy or ambiguous content.

Test short contexts too, since extending positional behavior can regress the original range. Include multiple distant dependencies rather than only one easily found marker, and vary where the answer-bearing material appears.

Compare the same model before and after the scaling change with a fixed inference setup. Passing a synthetic retrieval test supports a narrow capability; it does not establish reliable long-document reasoning or every application workload.

101. Why can a RoPE cache become invalid after changing the positional scaling configuration?

Cached keys may already encode rotations under the old positional convention. Queries computed with a new convention no longer interact with those keys as the model’s intended attention computation requires.

A configuration change can therefore require recomputing the prefix, even when token IDs and weights remain unchanged. Whether a runtime caches pre-rotation or post-rotation representations is an implementation detail that must be checked.

I would include positional configuration in cache compatibility checks and test a resumed request against full recomputation. Treating the cache as generic reusable token memory obscures the mathematical state it actually contains.

102. How would you validate a quantized KV cache separately from quantized weights?

I would hold the weights and other settings fixed, change only cache quantization and compare logits as context grows. Cache errors can accumulate or vary with token position in ways that a short weight-quantization benchmark misses.

Inspect scale layout, residual high-precision regions if used, and conversion at block boundaries. Evaluate long dependencies and repeated decoding steps, not just one prompt forward pass.

Measure memory savings and conversion overhead along with task quality. Cache quantization and weight quantization can interact, so after isolated tests the combined configuration needs qualification rather than assuming their errors add harmlessly.

103. What correctness risks arise when reusing CUDA graphs for dynamic request batches?

Captured execution expects compatible shapes, memory addresses and control flow under its supported contract. A dynamic server must map changing requests into that fixed structure without leaving stale indices, lengths or pointers.

Padding inactive slots is only safe if masks and outputs exclude them correctly. Reusing a graph while cache block tables refer to reclaimed storage can produce incorrect results even though the graph launches successfully.

I would test request arrival, cancellation, slot reuse and every supported bucket transition against an uncaptured reference. Also measure capture and fallback costs; a graph path that works only for the easiest steady-state batch is not a complete serving solution.

104. How do you reason about prefill/decode disaggregation without ignoring cache transfer?

Separating prefill and decode workers allows independent resource allocation, but decode needs the prefix state produced by prefill. Moving that state consumes network bandwidth, incurs latency and requires compatible layouts and ownership.

I would estimate cache bytes for the actual architecture and prompt distribution, then measure transfer overlap and queueing. A fast prefill worker does not help if transferring its output delays the first decode step.

Test failure during handoff and define which worker owns cancellation and cleanup. Compare total user latency and fleet cost against a colocated baseline; disaggregation is a workload-dependent system design, not an automatic improvement.

105. How would you make cache eviction and request cancellation safe under concurrent kernels?

I would ensure storage is not reclaimed while any in-flight kernel can still reference it. Logical cancellation and physical memory reuse are separate events, coordinated through the runtime’s stream and completion mechanisms.

Reference counts or ownership records must include shared prefixes and pending work. A request disappearing from the scheduler does not prove its device reads have finished.

Use stress tests that cancel and reuse slots at block boundaries while comparing surviving requests against a reference. This targets races that ordinary success-path tests miss. The design should specify when memory becomes reusable rather than rely on timing luck.

106. Why can identical seeded requests differ when batching changes?

Batching can alter random-number consumption, floating-point reduction order and kernel selection. A seed identifies an initial random state, but it does not specify every scheduling and numerical decision in the execution.

Per-request random streams can reduce dependence on neighboring requests, yet small logit differences can still change sampled or near-tied greedy tokens. After one token differs, later contexts are different.

I would define the reproducibility contract precisely: stable configuration and statistical behavior, request-local sampling reproducibility, or bitwise output. Test the promised level under batch composition changes instead of presenting a seed as a universal replay guarantee.

107. How would you validate speculative decoding when a grammar constrains allowed tokens?

The acceptance and correction logic must use the effective target and proposal distributions under the same applicable grammar state. Filtering only the final selected token can invalidate the distributional argument of the sampler.

Grammar state advances with committed tokens, so rejected draft suffixes must not leave it advanced. A token allowed after the draft history may be forbidden after a corrected token.

I would test a tiny grammar where exact sequence probabilities can be enumerated, including rejection and zero-probability cases. Compatibility between two individually correct features must be demonstrated; unconstrained speculative tests do not establish constrained correctness.

108. What can go wrong when temperature and top-p are applied in different orders?

Temperature changes relative probabilities, which can change the smallest cumulative-probability set selected by top-p. Filtering before temperature can therefore retain a different set of tokens than filtering after it.

An implementation comparison must include transformation order, tie handling and renormalization. Using identical numeric settings does not guarantee identical distributions when the pipeline differs.

from math import exp
scores = [2.0, 1.0, 0.0]
def probabilities(temperature):
    w = [exp((x - max(scores)) / temperature) for x in scores]
    return [v / sum(w) for v in w]
def nucleus_indices(probs, cutoff):
    chosen, mass = [], 0.0
    for i in sorted(range(len(probs)), key=probs.__getitem__, reverse=True):
        chosen.append(i)
        mass += probs[i]
        if mass >= cutoff:
            break
    return chosen
assert nucleus_indices(probabilities(1.0), 0.85) == [0, 1]
assert nucleus_indices(probabilities(2.0), 0.85) == [0, 1, 2]
print(nucleus_indices(probabilities(2.0), 0.85))  # [0, 1, 2]
from math import exp
scores = [2.0, 1.0, 0.0]
def probabilities(temperature):
    w = [exp((x - max(scores)) / temperature) for x in scores]
    return [v / sum(w) for v in w]
def nucleus_indices(probs, cutoff):
    chosen, mass = [], 0.0
    for i in sorted(range(len(probs)), key=probs.__getitem__, reverse=True):
        chosen.append(i)
        mass += probs[i]
        if mass >= cutoff:
            break
    return chosen
assert nucleus_indices(probabilities(1.0), 0.85) == [0, 1]
assert nucleus_indices(probabilities(2.0), 0.85) == [0, 1, 2]
print(nucleus_indices(probabilities(2.0), 0.85))  # [0, 1, 2]
from math import exp
scores = [2.0, 1.0, 0.0]
def probabilities(temperature):
    w = [exp((x - max(scores)) / temperature) for x in scores]
    return [v / sum(w) for v in w]
def nucleus_indices(probs, cutoff):
    chosen, mass = [], 0.0
    for i in sorted(range(len(probs)), key=probs.__getitem__, reverse=True):
        chosen.append(i)
        mass += probs[i]
        if mass >= cutoff:
            break
    return chosen
assert nucleus_indices(probabilities(1.0), 0.85) == [0, 1]
assert nucleus_indices(probabilities(2.0), 0.85) == [0, 1, 2]
print(nucleus_indices(probabilities(2.0), 0.85))  # [0, 1, 2]

I would calculate a small explicit example and compare support as well as probabilities. Document the configured order and preserve it in speculative or constrained paths, since acceptance logic must refer to the actual post-transformation distributions.

109. Why can an all-masked attention row produce NaNs, and what should define its behavior?

If all scores are negative infinity, a naive stable-softmax subtraction can form negative infinity minus negative infinity. The row has no valid probability distribution over keys, so an implementation needs an explicit policy.

For padding-only queries, the intended result may be a zeroed output that is ignored downstream. For a real query, an all-masked row may indicate a mask or position bug and should be surfaced rather than silently normalized.

I would test the chosen backend’s behavior and enforce the application invariant. Replacing infinities with a large finite negative value can accidentally create a uniform distribution over forbidden positions, which avoids NaNs while violating semantics.

110. How would you compare attention kernels when tiny logit differences become large generation differences?

I would first compare intermediate outputs and logits on the same fixed token history, using absolute and relative tolerances appropriate to dtype and magnitude. Free-running generations diverge after a token difference and are no longer aligned numerical tests.

Inspect whether discrepancies concentrate near ties, long reductions or unsupported masks. Then evaluate downstream task behavior separately to determine whether the numerical differences are acceptable for the deployment.

The conclusion should distinguish a kernel bug from expected floating-point variation. Bitwise mismatch is not automatically incorrect, but “the text looks fine” is not enough to dismiss a systematic error outside the stated tolerance.

111. How can an inference memory leak be distinguished from intentional allocator caching?

I would compare live tensor allocations, allocator-reserved memory and request/cache ownership over repeated bounded workloads. Reserved memory can remain high after tensors are freed because an allocator retains blocks for reuse.

A leak is suggested when live or unreclaimable state grows with completed requests, while a stable reserved plateau may be normal. Shared-prefix references, canceled requests and retained Python objects are useful places to inspect.

Use the framework’s memory tools and a controlled sequence that returns to the same workload state. Calling an empty-cache function repeatedly can obscure the pattern and reduce performance without fixing a retained reference or incorrect ownership lifecycle.

112. How would you diagnose performance regressions caused by dynamic-shape recompilation?

I would record compile events and correlate them with sequence lengths, batch sizes and control-flow changes. A warm benchmark on one shape can conceal repeated compilation under a diverse production distribution.

Compare shape bucketing, static bounds and an eager fallback while measuring both padding waste and compilation overhead. More buckets can reduce wasted compute but increase the number of compiled variants and memory consumption.

Test the real arrival distribution, including rare long requests. The best configuration depends on reuse frequency and latency requirements; disabling all dynamic behavior may produce a stable graph at an unacceptable memory or throughput cost.

113. Why can a faster tokenizer fail to improve end-to-end training throughput?

Tokenization may already be off the critical path, or its gains may be offset by storage, decompression, packing or data transfer. Throughput is limited by the slowest non-overlapped stage, not by the most visible CPU function.

I would measure worker queues, accelerator idle time and batch-ready latency before changing the pipeline. Pre-tokenization can help but adds artifact versioning and storage costs, and must preserve the exact tokenizer configuration.

Compare steady-state throughput and restart behavior. A pipeline that is fast after a long cache-building phase may be unsuitable for short experiments or frequently changing data. End-to-end measurements should include the phases relevant to the actual workflow.

114. How should a training data loader resume without silently repeating or skipping records?

It needs a recoverable sampling position and deterministic mapping from that position to records under the chosen shuffle and sharding policy. Worker-local offsets alone may be insufficient when workers prefetch or topology changes.

I would save dataset revision, shuffle seed or permutation state, consumed progress and any buffered-record policy. Distinguish fetched records from records whose gradients were included in a completed update.

Test interruption immediately before and after an optimizer step and compare the consumed record sequence. Some systems intentionally accept bounded replay; if so, document that semantic instead of calling it exact continuation.

115. What makes tokenizer-vocabulary expansion a model migration?

New token IDs require new or resized embedding and output entries, and tied weights must remain consistent. Existing text may segment differently if tokenizer rules change, so the migration can affect more than the newly added strings.

I would preserve old ID assignments, define initialization for new entries and train them with an appropriate objective. Check special-token semantics and generation masks as well as tensor shapes.

Validate old-task regression and new-token behavior before exporting. A loader accepting the larger vocabulary proves only structural compatibility; it does not prove the model learned useful meanings for the new entries or preserved the old distribution.

116. How would you estimate whether adapter multiplexing is better than separate merged replicas?

I would compare memory, request mix, batching compatibility and adapter-switch overhead. Sharing base weights can save memory, while separate merged replicas may simplify execution and offer efficient kernels for predictable high-volume traffic.

A long tail of rarely used adapters favors different trade-offs from a few equally busy adapters. Cache compatibility must include adapter identity because prior hidden states can depend on the adapter that produced them.

Benchmark realistic concurrency and isolation requirements, including cold adapter loading. The decision should use total served work at the latency target rather than only base-weight memory saved.

117. How can a model conversion preserve tensor shapes while corrupting attention?

Conversion can transpose, concatenate or permute projections incorrectly while retaining the same total dimensions. Q/K head ordering and rotary coordinate pairing are especially vulnerable because several layouts have compatible shapes.

I would compare layerwise outputs against the source model on fixed token fixtures, locating the first divergence. A checksum only proves file identity, and a successful load only proves the destination accepted its structure.

Test cache and multi-token behavior after basic forward parity. Some errors are hidden in single-token inputs where causal or positional distinctions are trivial. Preserve the conversion script and source configuration so the mapping is reviewable.

118. What should a minimal reference implementation sacrifice to remain useful?

It should favor explicit shapes, masks and arithmetic over speed, while implementing the same defined operation as the optimized path. Its value is clarity and independent structure, not production throughput.

For attention, a small dense implementation can materialize scores and probabilities so each intermediate is inspectable. It should use inputs small enough to avoid memory pressure and a precision suitable for the intended comparison.

Keep the reference independent of the optimized code’s indexing shortcuts; otherwise both can share the same bug. State unsupported features explicitly and use it to validate bounded invariants rather than claim it represents the entire production stack.

119. How would you prioritize tests for a new distributed language-model training stack?

I would begin with single-device objective and gradient correctness, then add one distributed feature at a time. Compare each stage with a reference before combining sharding, accumulation, mixed precision and checkpointing.

Prioritize boundaries that can silently change the objective: target alignment, token normalization, collective averaging, optimizer-state mapping and resume position. Then exercise failures such as uneven input exhaustion and interrupted checkpoint writes.

Performance tests follow a correct baseline and include realistic shapes and topology. A fast run with a plausible loss curve is insufficient because systematic normalization or data-replay errors can remain invisible until expensive downstream evaluation.

120. What would make you reject an apparently successful LLM optimization experiment?

I would reject the conclusion if the comparison changed the objective, data, precision contract or workload without accounting for it. I would also reject gains based on incomplete outputs, omitted slow requests or a reference configuration that was not properly warmed and tuned.

For example, a faster attention path that silently drops old context is a model-behavior change, while a training speedup that counts padded tokens as useful throughput inflates the claimed gain. Both can produce attractive charts without solving the intended problem.

A defensible result states the invariant preserved, the measured benefit, the tested configurations and remaining limitations. Keep correctness fixtures, profiler evidence and reproducible artifacts together so another engineer can distinguish an implementation improvement from a changed experiment.

Large Language Models Documentation and Further Reading

These primary papers and official resources support the architecture and engineering discussion. Paper results apply to their experimental settings, and evolving framework documentation must be matched to the installed version. The local numerical examples do not establish distributed-training correctness, model convergence or accelerator performance.

Team Sherlock.sh

Building fraud intelligence for live high-stake video interactions (financial discussions, hiring interviews, board meeting etc).

Catch candidate fraud before it costs you.

Sherlock AI helps you detect AI-assisted, proxy, and deepfake candidates, so every hiring decision is backed by evidence.

Ask AI about Sherlock.sh

Similar Posts