Interviews for roles that build with large language models rarely stop at “what is a transformer?”. They follow the life of a real feature: choose a model, shape the prompt, ground it in data, check the output, secure it and keep it affordable. This guide is organised the same way. Each section gives the questions you are likely to hear and a model answer you can adapt.
The model answers are illustrative. They show the shape and depth an interviewer expects; replace the examples with your own projects.
Stage 1: Foundations
Q1. In plain terms, what is a transformer and why did it matter?
Model answer (illustrative): “A transformer is a neural network architecture built around attention, which lets every token in a sequence weigh its relationship to every other token. The 2017 paper that introduced it, ‘Attention Is All You Need’, described a model based solely on attention mechanisms, dispensing with recurrence and convolutions. Because it does not process tokens one after another like a recurrent network, training is far more parallelisable, which is a large part of why models could be scaled up.”
Q2. What are tokens and a context window, and why should a product team care?
Model answer (illustrative): “Models read and write text as tokens, which are pieces of words. The context window is the maximum number of tokens the model can consider in one request, covering the instructions, any retrieved documents, the conversation so far and the answer. It matters because cost and latency usually scale with tokens, and because anything that does not fit is simply invisible to the model.”
Q3. What is a hallucination, and why does it happen?
Use precise language here. NIST’s Generative AI Profile (NIST AI 600-1, July 2024) calls this risk confabulation: generative systems producing and confidently presenting erroneous or false content, colloquially known as hallucinations or fabrications.
Model answer (illustrative): “A language model generates likely continuations of text, not verified facts. When the training data is thin, out of date or the question needs private information, it can still produce something fluent and wrong. I reduce the risk by grounding answers in retrieved sources, asking the model to cite them, allowing it to say it does not know, and checking outputs in evaluation.”
Stage 2: Shaping behaviour
Q4. Prompting, RAG or fine-tuning: how do you choose?
| Approach | Best when | Watch out for |
|---|---|---|
| Prompting | The model already has the knowledge; you need format, tone or steps | Long prompts raise cost; behaviour can shift between model versions |
| Retrieval-augmented generation | Answers depend on private, changing or citable documents | Retrieval quality limits answer quality |
| Fine-tuning | You need a consistent style or task behaviour that prompting cannot achieve reliably | Needs good training data, evaluation and re-training as needs change |
This table reflects common practice rather than a single authority. A strong answer says you start with the cheapest option, measure, and only move right when the evaluation shows a gap.
Q5. How do you write a prompt that holds up in production?
Model answer (illustrative): “I separate fixed instructions from user input, state the task, the audience and the output format explicitly, give one or two examples, and tell the model what to do when it lacks information. If the output feeds code, I ask for a structured format such as JSON and validate it before use. Then I version the prompt and run it against a test set every time it changes.”
Stage 3: Grounding with your own data
Q6. Explain retrieval-augmented generation.
Know the origin. Lewis and colleagues’ 2020 paper combined parametric memory, a pre-trained sequence-to-sequence model, with non-parametric memory, a dense vector index of Wikipedia accessed by a neural retriever. They reported more specific, diverse and factual generation than a parametric-only baseline.
Model answer (illustrative): “At indexing time, we split documents into chunks, turn each chunk into an embedding and store it in a vector index with its metadata. At query time, we embed the question, retrieve the most similar chunks, optionally re-rank them, and pass them to the model with an instruction to answer only from that context and cite it.”
Q7. Answers from your RAG system are wrong. How do you debug it?
Interviewers want to see you split the problem in two:
- Was the right content retrieved? Inspect the retrieved chunks for failing queries. If the answer is not there, look at chunk size, overlap, metadata filters, the embedding model and whether keyword search should be combined with vector search.
- If it was retrieved, did the model use it? Check whether the context was too long, contradictory or badly ordered, and whether the instruction to stay within the sources is clear.
- Is the source itself wrong or stale? Fix the data pipeline, not the prompt.
Stage 4: Proving it works
Q8. How do you evaluate an LLM feature?
Model answer (illustrative): “I build a test set from real or realistic inputs, including edge cases and known failures, with expected answers or grading criteria. I score automatically where I can, for example exact match on structured fields, and use human review or a carefully checked model-as-grader for open-ended answers. I track groundedness, correctness, format validity, refusal behaviour, latency and cost, and I rerun the suite on every prompt, model or retrieval change. After launch, I sample live traffic and collect user feedback.”
Freshers can score well here simply by showing they test at all. Many candidates describe a demo and no evaluation.
Stage 5: Security and risk
Q9. What is prompt injection and how do you limit it?
OWASP’s 2025 Top 10 for LLM applications lists Prompt Injection first (LLM01:2025). The full list is worth memorising because interviewers often use it as a checklist:
- LLM01 Prompt Injection; LLM02 Sensitive Information Disclosure; LLM03 Supply Chain; LLM04 Data and Model Poisoning; LLM05 Improper Output Handling
- LLM06 Excessive Agency; LLM07 System Prompt Leakage; LLM08 Vector and Embedding Weaknesses; LLM09 Misinformation; LLM10 Unbounded Consumption
OWASP’s Prompt Injection Prevention Cheat Sheet describes indirect injection as malicious instructions hidden in external content the model processes, such as web pages and documents it fetches, and recommends least-privilege permissions, human approval for consequential tool actions, monitoring outputs for signs of injection and regular testing with known attack patterns.
Model answer (illustrative): “Prompt injection is when input, typed by a user or hidden in a document or web page the model reads, changes the model’s behaviour against the developer’s intent. There is no single fix, so I layer defences: treat all model output as untrusted and validate it before acting on it, give tools and agents the minimum permissions they need, require human approval for high-impact actions, keep secrets out of prompts, and test with adversarial inputs.”
Q10. What other risks would you raise with a product owner?
NIST AI 600-1 lists twelve risks that generative AI creates or makes worse, including confabulation, data privacy, harmful bias or homogenisation, information integrity, information security, intellectual property and value chain and component integration. Pick the two or three that fit the product in the interview, and say what control you would add for each.
Stage 6: Running it in production
Q11. Design a support assistant that answers from 50,000 help articles.
Treat this as a structured walk-through rather than a single answer:
- Clarify: languages, users, acceptable latency, what happens when the assistant is unsure, and whether it can take actions or only answer.
- Ingest: clean and chunk articles, store embeddings with metadata such as product and date, re-index on change.
- Serve: retrieve, re-rank, generate with citations, validate the format, fall back to a human or a search results page on low confidence.
- Guard: filter inputs and outputs, limit request size and rate to control unbounded consumption, log safely without storing personal data unnecessarily.
- Measure: offline test set before launch; resolution rate, escalation rate, feedback, latency and cost per conversation after launch.
- Optimise: cache frequent answers, use a smaller model for simple routing, and trim context to what is needed.
Q12. The feature is too slow and too expensive. What do you try first?
Model answer (illustrative): “First I measure where tokens and time go. Then, in rough order: shorten prompts and retrieved context, cache repeated requests, stream responses so users see progress, route easy requests to a smaller model, and batch offline work. I check each change against the evaluation set so I do not trade quality for speed without noticing.”
Freshers versus experienced candidates
Freshers are judged mainly on Stages 1 to 3 and on one well-explained project. Experienced candidates are expected to lead with Stages 4 to 6: evaluation, risk and operations, using numbers from systems they have run.
Whatever your level, list the concrete pieces on your CV: models and frameworks used, the data you grounded on and how you measured quality. A software engineer resume builder helps structure project entries, and if your work leans on analytics, compare with the data analyst resume builder. Before you apply, check keyword coverage with the ATS resume score checker.
Frequently asked questions
What generative AI interview questions do freshers get?
Mostly foundations: what a token and a context window are, how prompting differs from fine-tuning, what RAG is and why models hallucinate. Expect to walk through a small project, such as a chatbot over your own documents, and explain how you tested it.
What do experienced candidates get asked?
System design for an LLM feature, evaluation strategy, cost and latency trade-offs, security threats such as prompt injection, and how they monitored a model-backed feature after launch.
Do I need to know the maths behind transformers?
For applied LLM engineering roles, a clear conceptual explanation of attention and tokens is usually enough. Research and model-training roles go much deeper, so check the job description.
Should I name specific model providers in my answers?
Name the ones you have actually used, and explain why you chose them. Avoid claims about which model is best; interviewers care more about how you would compare options for their use case.
Sources
- Attention Is All You Need (Vaswani et al., 2017), arXiv:1706.03762 (arXiv)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), arXiv:2005.11401 (arXiv)
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024) (National Institute of Standards and Technology)
- OWASP Top 10 for LLM Applications 2025 (OWASP Gen AI Security Project)
- LLM Prompt Injection Prevention Cheat Sheet (OWASP Cheat Sheet Series)
- OWASP Top 10 for Large Language Model Applications (project page) (OWASP Foundation)