LLM Prompting & Testing

If you're looking to master prompt engineering, AI model behavior, and build testable, production-grade prompts, this is your field guide. This guide walks through everything from basic prompting anatomy to evaluations (Evals) and parameter tuning.

---

What is Prompt Engineering?

Prompt engineering is the process of crafting instructions to guide large language models (LLMs) in generating reliable, relevant, and safe responses.

A "prompt" is any input or instruction you send to the model.

---

LLM Request Structure – API Anatomy

Example Request (OpenAI SDK - Python):

pythonCopyEditfrom openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-4.1",
    input="Write a one-sentence bedtime story about a unicorn."
)

print(response.output_text)

Key Parameters:

---

Message Roles in Chat Models

These are not prompt types, but how the API structures the interaction.

---

Prompt Types (Techniques)

---

Prompt Writing Best Practices

Do:

• Be explicit about your expectations.

• Use examples (few-shot) when accuracy matters.

• Use system role for behavior shaping.

• Break tasks into steps (CoT) for logic problems.

Avoid:

• Vague instructions: "Explain this?"

• Overloading the prompt with unrelated info.

• Relying on temperature alone to fix response quality.

---

Temperature vs Top_p

For stable, accurate outputs, set temperature=0.2 and top_p=0.9 as a starting point.

---

Summary Cheat Sheet

---

What is RAG?

Retrieval-Augmented Generation (RAG) is a technique that combines two powerful components:

1. Retrieval system: This searches a large external database or document collection to find relevant information based on a user's question.

2. Generative LLM (Large Language Model): This uses the retrieved information to generate a precise, context-aware response

How RAG uses prompts

• Normally, when you prompt an LLM, you send it a question or instruction, and it tries to answer based only on what it "knows" internally.

• In RAG, before generating an answer, the system retrieves relevant documents related to your question.

• These documents are then added into the prompt sent to the LLM.

Example:

User question:
“What is the company policy on remote work?”

1. The retrieval system searches your company’s internal documents and finds a policy document about remote work.

2. The prompt to the LLM becomes something like:

   Based on the following document excerpts, answer the question.

   Document excerpts:

   • "Our company allows remote work up to 3 days a week."

   • "Remote work requests should be approved by the team lead."

Question: What is the company policy on remote work?

What's Next: Model Evaluation

Prompting is just the start. In a production-grade system, you’ll also:

• Use Promptfoo or OpenAI Evals to measure:

  • Consistency

  • Accuracy

  • Regression (across model versions)

• Save prompt snapshots and logs

• Write test cases in YAML/JSON for automated evaluation

---

What next: