tutorials/rag/evaluate-rag-flow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c5b81e0e5be92583",
   "metadata": {},
   "source": [
    "# Humanloop RAG Evaluation Walkthrough\n",
    "The goal of this notebook is to demonstrate how to take an existing RAG pipeline and integrate Humanloop in order to:\n",
    "1. Setup logging for your AI application using a Flow.\n",
    "2. Create a [Dataset](https://humanloop.com/docs/v5/concepts/datasets) and run Evaluations to benchmark the performance of your RAG pipeline.\n",
    "3. Extend your logging to include Prompt and Tool steps within your Flow.\n",
    "\n",
    "\n",
    "## What is Humanloop?\n",
    "Humanloop is an interactive development environment designed to streamline the entire lifecycle of LLM app development. It serves as a central hub where AI, Product, and Engineering teams can collaborate on Prompt management, Evaluation and Monitoring workflows. \n",
    "\n",
    "\n",
    "## What is RAG?\n",
    "RAG stands for Retrieval Augmented Generation.\n",
    "- **Retrieval** - Getting the relevant information from a larger data source for a given a query.\n",
    "- **Augmented** - Using the retrieved information as input to an LLM.\n",
    "- **Generation** - Generating an output from the model given the input.\n",
    "\n",
    "In practise, it remains an effective way to exploit LLMs for things like question answering, summarization, and more, where the data source is too large to fit in the context window of the LLM, or where providing the full data source for each query is not cost-effective.\n",
    "\n",
    "\n",
    "## What are the major challenges with RAG?\n",
    "Implementing RAG and other similar flows complicates the process of [Prompt Engineering](https://humanloop.com/blog/prompt-engineering-101) because you expand the design space of your application. There are lots of choices you need to make around the retrieval and Prompt components that can significantly impact the performance of your overall application. For example,\n",
    "- How do you select the data source?\n",
    "- How should it be chunked up and indexed?\n",
    "- What embedding and retrieval model should you use?\n",
    "- How should you combine the retrieved information with the query?\n",
    "- What should your system Prompt be? \n",
    "- Which model should you use?\n",
    "- What should your system message be?\n",
    "etc...\n",
    "\n",
    "The process of versioning, evaluating and monitoring your pipeline therefore needs to consider both the retrieval and generation components. This is where Humanloop can help.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf4a5f234d7bac09",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "53965cd45425bab5",
   "metadata": {},
   "source": [
    "# Example RAG Pipeline\n",
    "\n",
    "We first need a reference RAG implementation. Our use case will be Q&A over a corpus of medical documents.\n",
    "\n",
    "- **Dataset**: we'll use a version of the [MedQA dataset](https://huggingface.co/datasets/bigbio/med_qa) from Hugging Face. This is a multiple choice question answering problem based on the United States Medical License Exams (USMLE), with reference text books that contain the required information to answer the questions.\n",
    "- **Retriever**: we're going to use [Chroma](https://docs.trychroma.com/getting-started) as a simple local vector DB with their default embedding model `all-MiniLM-L6-v2`. You can replace this with your favorite retrieval system.\n",
    "- **Prompt**: **the Prompt will be managed in code**, populated with the users question and the context retrieved from the Retriever and sent to [OpenAI](https://platform.openai.com/docs/api-reference/introduction) to generate the answer.\n",
    "\n",
    "### Where to store your Prompts?\n",
    "\n",
    "Generally speaking, when the engineering/applied AI teams are mainly responsible for managing the details of the Prompt, then the pattern of storing or constructing the Prompt in code works well. This is the pattern we follow in this tutorial. \n",
    "\n",
    "However, if the Product/Domain Expert teams are more involved in Prompt engineering and management, then the Prompt can instead be managed on Humanloop and retrieved or called by your code - this workflow lies outside the scope of this tutorial and we cover it separately. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da007e8494c60446",
   "metadata": {},
   "source": [
    "## Complete Prerequisites\n",
    "\n",
    "### Install packages\n",
    "We use poetry to manage dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5b3e3f74731b50ab",
   "metadata": {
    "jupyter": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mInstalling dependencies from lock file\u001b[39m\n",
      "\n",
      "No dependencies to install or update\n"
     ]
    }
   ],
   "source": [
    "!poetry install"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80f74709ea0177a8",
   "metadata": {},
   "source": [
    "### Initialise the SDKs\n",
    "\n",
    "You will need to set your OpenAI API key in the  `.env` file in the root of the repo. You can retrieve your API key from your [OpenAI account](https://platform.openai.com/api-keys).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "47ac94aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up dependencies\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "from chromadb import chromadb\n",
    "from openai import OpenAI\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "# load .env file that contains API keys\n",
    "load_dotenv()\n",
    "\n",
    "# init clients\n",
    "chroma = chromadb.Client()\n",
    "openai = OpenAI(api_key=os.getenv(\"OPENAI_KEY\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f74ffee245036bc",
   "metadata": {},
   "source": [
    "### Set up the Vector DB\n",
    "This involves loading the data from the MedQA dataset and embedding the data within a collection in Chroma. This will take a couple of minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "15c5158d1d159535",
   "metadata": {},
   "outputs": [],
   "source": [
    "# init collection into which we will add documents\n",
    "collection = chroma.get_or_create_collection(name=\"MedQA\")\n",
    "\n",
    "# load knowledge base\n",
    "knowledge_base = pd.read_parquet(\"../../assets/sources/textbooks.parquet\")\n",
    "knowledge_base = knowledge_base.sample(10, random_state=42)\n",
    "\n",
    "\n",
    "# Add to Chroma - will by default use local vector DB and model all-MiniLM-L6-v2\n",
    "collection.add(\n",
    "    documents=knowledge_base[\"contents\"].to_list(),\n",
    "    ids=knowledge_base[\"id\"].to_list(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd43484af0f2f10c",
   "metadata": {},
   "source": [
    "### Define the Prompt\n",
    "We define a simple prompt template that has variables for the question, answer options and retrieved data.\n",
    "\n",
    "It is generally good practise to define the Prompt details that impact the behaviour of the model in one place separate to your application logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "187af8c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"gpt-4o\"\n",
    "temperature = 0\n",
    "template = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"\"\"Answer the following question factually.\n",
    "\n",
    "Question: {{question}}\n",
    "\n",
    "Options:\n",
    "- {{option_A}}\n",
    "- {{option_B}}\n",
    "- {{option_C}}\n",
    "- {{option_D}}\n",
    "- {{option_E}}\n",
    "\n",
    "---\n",
    "\n",
    "Here is some retrieved information that might be helpful.\n",
    "Retrieved data:\n",
    "{{retrieved_data}}\n",
    "\n",
    "---\n",
    "\n",
    "Give you answer in 3 sections using the following format. Do not include the quotes or the brackets. Do include the \"---\" separators.\n",
    "```\n",
    "<chosen option verbatim>\n",
    "---\n",
    "<clear explanation of why the option is correct and why the other options are incorrect. keep it ELI5.>\n",
    "---\n",
    "<quote relevant information snippets from the retrieved data verbatim. every line here should be directly copied from the retrieved data>\n",
    "```\n",
    "\"\"\",\n",
    "    }\n",
    "]\n",
    "\n",
    "def populate_template(template: list, inputs: dict[str, str]) -> list:\n",
    "    \"\"\"Populate a template with input variables.\"\"\"\n",
    "    messages = []\n",
    "    for i, template_message in enumerate(template):\n",
    "        content = template_message[\"content\"]\n",
    "        for key, value in inputs.items():\n",
    "            content = content.replace(\"{{\" + key + \"}}\", value)\n",
    "        message = {**template_message, \"content\": content}\n",
    "        messages.append(message)\n",
    "    return messages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e86410ee6d885c4b",
   "metadata": {},
   "source": [
    "## Define the RAG Pipeline\n",
    "\n",
    "Now we provide the reference RAG pipeline using Chroma and OpenAI that takes a question and returns an answer. This is ultimately what we will evaluate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "53c95ad9790ade59",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai.types.chat import ChatCompletionMessage\n",
    "\n",
    "def retrieval_tool(question: str) -> str:\n",
    "    \"\"\"Retrieve most relevant document from the vector db (Chroma) for the question.\"\"\"\n",
    "    response = collection.query(query_texts=[question], n_results=1)\n",
    "    retrieved_doc = response[\"documents\"][0][0]\n",
    "    return retrieved_doc\n",
    "\n",
    "def ask_question(inputs: dict[str, str])-> ChatCompletionMessage:\n",
    "    \"\"\"Ask a question and get an answer using a simple RAG pipeline\"\"\"\n",
    "    \n",
    "    # Retrieve context\n",
    "    retrieved_data = retrieval_tool(inputs[\"question\"])\n",
    "    inputs = {**inputs, \"retrieved_data\": retrieved_data}\n",
    "    \n",
    "    # Populate the Prompt template\n",
    "    messages = populate_template(template, inputs)\n",
    "    \n",
    "    # Call OpenAI to get response\n",
    "    chat_completion = openai.chat.completions.create(\n",
    "        model=model,\n",
    "        temperature=temperature,\n",
    "        messages=messages,\n",
    "    )\n",
    "    return chat_completion.choices[0].message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da57f2c4b4533a09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the pipeline\n",
    "chat_completion = ask_question(\n",
    "    {\n",
    "        \"question\": \"A 34-year-old male suffers from inherited hemophilia A. He and his wife have three unaffected daughters. What is the probability that the second daughter is a carrier of the disease?\",\n",
    "        \"option_A\": \"0%\",\n",
    "        \"option_B\": \"25%\",\n",
    "        \"option_C\": \"50%\",\n",
    "        \"option_D\": \"75%\",\n",
    "        \"option_E\": \"100%\",\n",
    "    }\n",
    ")\n",
    "print(chat_completion.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9fc3cffe1b5cb99",
   "metadata": {},
   "source": [
    "# Humanloop Integration\n",
    "\n",
    "We now integrate Humanloop into the RAG pipeline to first enable logging and then to trigger evaluations against a dataset.\n",
    "\n",
    "\n",
    "## Initialise the SDK\n",
    "You will need to set your Humanloop API key in the  `.env` file in the root of the repo. You can retrieve your API key from your [Humanloop organization](https://app.humanloop.com/account/api-keys).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c2102bcad49c932",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Init the Humanloop SDK\n",
    "from humanloop import Humanloop\n",
    "\n",
    "load_dotenv()\n",
    "humanloop = Humanloop(api_key=os.getenv(\"HUMANLOOP_KEY\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d598fd",
   "metadata": {},
   "source": [
    "## Integrate Logging\n",
    "\n",
    "Below, we just need to add a `humanloop.flows.log(...)` call after we execute our pipeline. We include in the `attributes` the information that will version our Flow on Humanloop. This can be arbitrary configuration of your choice.\n",
    "\n",
    "On running this updated code, Humanloop will now begin to track the versions of your Flow along with inputs, outputs and associated metadata. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "acfe1f5d648b980",
   "metadata": {
    "jupyter": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CreateFlowLogResponse(id='log_LscHgTkvAUcGf0g47xV5o', flow_id='fl_jLavsElEpb6dqckBMRObh', version_id='flv_ebvvb8oK8i3UmDZnYkQ6X', trace_status=None, status='complete')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add a `humanloop.prompts.log()` call after `ask_question()` to log the response and Prompt details to your Humanloop Prompt\n",
    "import inspect \n",
    "from datetime import datetime\n",
    "\n",
    "inputs = {\n",
    "    \"question\": \"A 34-year-old male suffers from inherited hemophilia A. He and his wife have three unaffected daughters. What is the probability that the second daughter is a carrier of the disease?\",\n",
    "    \"option_A\": \"0%\",\n",
    "    \"option_B\": \"25%\",\n",
    "    \"option_C\": \"50%\",\n",
    "    \"option_D\": \"75%\",\n",
    "    \"option_E\": \"100%\",\n",
    "}\n",
    "\n",
    "start_time = datetime.now()\n",
    "chat_completion = ask_question(inputs)\n",
    "\n",
    "# Log the response to a Humanloop Flow\n",
    "humanloop.flows.log(\n",
    "    path=\"evals_demo/medqa-flow\",\n",
    "    # We want our Prompt details and Tool implementation to define the version of our Flow. \n",
    "    # If these details change in code, Humanloop will bump the version automatically.\n",
    "    flow={\n",
    "        \"attributes\": {\n",
    "            \"prompt\": {\n",
    "                \"template\": template,\n",
    "                \"model\": model,\n",
    "                \"temperature\": temperature,\n",
    "            },\n",
    "            \"tool\": {\n",
    "                \"name\": \"retrieval_tool_v3\",\n",
    "                \"description\": \"Retrieval tool for MedQA.\",\n",
    "                \"source_code\": inspect.getsource(retrieval_tool),\n",
    "            },\n",
    "        }\n",
    "    },\n",
    "    inputs=inputs,\n",
    "    output=chat_completion.content,\n",
    "    start_time=start_time,\n",
    "    end_time=datetime.now(),\n",
    "    # We don't intend on adding any more Logs to our trace, so mark as complete.\n",
    "    trace_status=\"complete\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "696b87c33a608ed5",
   "metadata": {},
   "source": [
    "## Check your Humanloop workspace\n",
    "\n",
    "After running this pipeline, you will now see your Flow logs in your Humanloop workspace.\n",
    "\n",
    "If you make changes to your `attributes` in code and re-run the pipeline, you will see a new version of the Flow created in Humanloop.\n",
    "\n",
    "\n",
    "![Flow Logs](../../assets/images/flow_log.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "595e5f31dc65ba72",
   "metadata": {},
   "source": [
    "# Trigger Evaluations\n",
    "\n",
    "We will now extend our implementation to allow us to run Evaluations on Humanloop against a specific test dataset.\n",
    "\n",
    "This involves the following steps:\n",
    "1. Create a Dataset that we can manage and re-use on Humanloop as the source of truth.\n",
    "2. Create some Evaluators that we can manage and re-use on Humanloop that can provide judgements on the performance of our Pipeline.\n",
    "3. Trigger an Evaluation with logging to Humanloop.\n",
    "4. View the results.\n",
    "\n",
    "Then as you tweak your pipeline in code, this will allow you to easily track and compare the performance of different versions. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78969c2f",
   "metadata": {},
   "source": [
    "## Create a Dataset\n",
    "Here we will create a Dataset on Humanloop using the MedQA test dataset. Alternatively you can create a data from Logs on Humanloop, or upload via the UI - see our [guide](https://humanloop.com/docs/v5/evaluation/guides/create-dataset). \n",
    "\n",
    "You can then effectively version control your Dataset centrally on Humanloop and hook into it for Evaluation workflows in code and via the UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "87c70e0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_dataset_to_humanloop():\n",
    "    df = pd.read_json(\"../../assets/datapoints.jsonl\", lines=True)\n",
    "\n",
    "    datapoints = [row.to_dict() for _i, row in df.iterrows()][0:20]\n",
    "    return humanloop.datasets.upsert(\n",
    "        path=\"evals_demo/medqa-test\",\n",
    "        datapoints=datapoints,\n",
    "        commit_message=f\"Added {len(datapoints)} datapoints from MedQA test dataset.\",\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "deae78c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = upload_dataset_to_humanloop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ffb5c76",
   "metadata": {},
   "source": [
    "## Set up Evaluators\n",
    "\n",
    "Here we will upload some Evaluators defined in code in `assets/evaluators/` so that Humanloop can manage running these for Evaluations (and later for Monitoring!)\n",
    "\n",
    "Alternatively you can define AI, Code and Human based Evaluators via the UI - see the relevant `How-to guides` on [Evaluations](https://humanloop.com/docs/v5/evaluation/overview) for creating Evaluators of different kinds.\n",
    "\n",
    "Further you can choose to not host the Evaluator on Humanloop and instead use your own runtime and instead post the results as part of the Evaluation. This can be useful for more complex workflows that require custom dependencies or resources. See the \"Running Evaluators locally\" section below for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0dcb9069",
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_evaluators():\n",
    "    # Upload Code Evaluators\n",
    "    for evaluator_name, return_type in [\n",
    "        (\"exact_match\", \"boolean\"),\n",
    "        (\"levenshtein\", \"number\"),\n",
    "    ]:\n",
    "        with open(f\"../../assets/evaluators/{evaluator_name}.py\", \"r\") as f:\n",
    "            code = f.read()\n",
    "        humanloop.evaluators.upsert(\n",
    "            path=f\"evals_demo/{evaluator_name}\",\n",
    "            spec={\n",
    "                \"evaluator_type\": \"python\",\n",
    "                \"arguments_type\": \"target_required\",\n",
    "                \"return_type\": return_type,\n",
    "                \"code\": code,\n",
    "            },\n",
    "            commit_message=f\"New version from {evaluator_name}.py\",\n",
    "        )\n",
    "\n",
    "    # Upload an LLM Evaluator\n",
    "    humanloop.evaluators.upsert(\n",
    "        path=\"evals_demo/reasoning\",\n",
    "        spec={\n",
    "            \"evaluator_type\": \"llm\",\n",
    "            \"arguments_type\": \"target_free\",\n",
    "            \"return_type\": \"boolean\",\n",
    "            \"prompt\": {\n",
    "                \"model\": \"gpt-4o\",\n",
    "                \"endpoint\": \"complete\",\n",
    "                \"temperature\": 0,\n",
    "                \"template\": \"An answer is shown below. The answer contains 3 sections, separated by \\\"---\\\". The first section is the final answer. The second section is an explanation. The third section is a citation.\\n\\nEvaluate if the final answer follows from the citation and the reasoning in the explanation section. Give a brief explanation/discussion. Do not make your judgment based on factuality, but purely based on the logic presented.\\nOn a new line, give a final verdict of \\\"True\\\" or \\\"False\\\".\\n\\nAnswer:\\n{{log.output}}\",\n",
    "            },\n",
    "        },\n",
    "        commit_message=\"Initial reasoning evaluator.\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8138e15d",
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_evaluators()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b48f645",
   "metadata": {},
   "source": [
    "## Run Evaluation\n",
    "\n",
    "Now we can start to trigger Evaluations on Humanloop using our Dataset and Evaluators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09ccddff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "12405c95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation created: evr_hlf4paqOeAy41ECJRtchp\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "# Create the Evaluation specifying the Dataset and Evaluators to use\n",
    "evaluation = humanloop.evaluations.create(\n",
    "    name=\"Demo evals 2\",\n",
    "    file={\"path\":\"evals_demo/medqa-flow\"},\n",
    "    evaluators=[\n",
    "        {\"path\": \"evals_demo/exact_match\"},\n",
    "        {\"path\": \"evals_demo/levenshtein\"},\n",
    "        {\"path\": \"evals_demo/reasoning\"},\n",
    "    ],\n",
    ")\n",
    "print(f\"Evaluation created: {evaluation.id}\")\n",
    "\n",
    "run = humanloop.evaluations.create_run(\n",
    "    id=evaluation.id,\n",
    "    dataset={\"path\": \"evals_demo/medqa-test\"},\n",
    ")\n",
    "print(f\"Run created: {run.id}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c361cda6",
   "metadata": {},
   "source": [
    "Add `source_datapoint_id` and `evaluation_id` to the `humanloop.flow.log(...)` so that the logs are added the Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9b36091812b178d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def populate_evaluation():\n",
    "    \"\"\"Run a variation of your Pipeline over the Dataset to populate results\"\"\"\n",
    "    retrieved_dataset = humanloop.datasets.get(\n",
    "        id=run.dataset.id,\n",
    "        include_datapoints=True,\n",
    "    )\n",
    "    for datapoint in tqdm(retrieved_dataset.datapoints):\n",
    "        start_time = datetime.now()\n",
    "        \n",
    "        chat_completion = ask_question(datapoint.inputs)\n",
    "        \n",
    "        humanloop.flows.log(\n",
    "            path=\"evals_demo/medqa-flow\",\n",
    "            flow={\n",
    "                \"attributes\": {\n",
    "                    \"prompt\": {\n",
    "                        \"template\": template,\n",
    "                        \"model\": model,\n",
    "                        \"temperature\": temperature,\n",
    "                    },\n",
    "                    \"tool\": {\n",
    "                        \"name\": \"retrieval_tool_v4\",\n",
    "                        \"description\": \"Retrieval tool for MedQA.\",\n",
    "                        \"source_code\": inspect.getsource(retrieval_tool),\n",
    "                    },\n",
    "                }\n",
    "            },\n",
    "            inputs=inputs,\n",
    "            output=chat_completion.content,\n",
    "            start_time=start_time,\n",
    "            end_time=datetime.now(),\n",
    "            trace_status=\"complete\",\n",
    "            # NB: New arguments to link to Evaluation and Dataset\n",
    "            source_datapoint_id=datapoint.id,\n",
    "            run_id=run.id,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c951317c",
   "metadata": {},
   "outputs": [],
   "source": [
    "populate_evaluation()\n",
    "\n",
    "# Then change your pipeline and run this function again, keeping the Evaluation ID the same, to populate additional columns in your Evaluation!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4dd1e7b7e542c45",
   "metadata": {},
   "source": [
    "## Get Results and URL\n",
    "We can not get the aggregate results via the API and the URL to navigate to the Evaluation in the Humanloop UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e22e0558dc082de6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL:  https://stg.humanloop.com/project/fl_jLavsElEpb6dqckBMRObh/evaluations/evr_hlf4paqOeAy41ECJRtchp/stats\n"
     ]
    }
   ],
   "source": [
    "evaluation = humanloop.evaluations.get(id=evaluation.id)\n",
    "print(\"URL: \", evaluation.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c8dc0c8-7cb9-41f4-925f-caca59f717d1",
   "metadata": {},
   "source": [
    "![Flow Evals](../../assets/images/flow_evals.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e2617b1-1374-42dc-be13-4aa9414d1512",
   "metadata": {},
   "source": [
    "## Logging the full trace\n",
    "\n",
    "So far our Humanloop integration only cares about the output of our RAG pipeline. In many applications, it's helpful to have visibility into the behaviour of the components that make up your pipeline. In this example, we have a retrieval step and a LLM step that make up our application.\n",
    "\n",
    "We'll now demonstrate how to extend your Humanloop logging with more fidelity; adding separate Tool and Prompt steps into your Flow logs to give you full visibility. \n",
    "\n",
    "We add additional logging steps to our `ask_pipeline` that are linked to our Flow Log to represent the full trace of events on Humanloop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8838d1dd-a7c3-49d4-80f7-90bde6d983f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want to nest our subsequent logs under our Flow log to represent the full trace of events.\n",
    "\n",
    "def ask_question(inputs: dict[str, str], trace_id: str)-> ChatCompletionMessage:\n",
    "    \"\"\"Ask a question and get an answer using a simple RAG pipeline\"\"\"\n",
    "    \n",
    "    # Retrieve context\n",
    "    start_time=datetime.now()\n",
    "    retrieved_data = retrieval_tool(inputs[\"question\"])\n",
    "    inputs = {**inputs, \"retrieved_data\": retrieved_data}\n",
    "\n",
    "    # Log the retriever information to Humanloop separately\n",
    "    humanloop.tools.log(\n",
    "        path=\"evals_demo/medqa-retrieval\",\n",
    "        tool={\n",
    "            \"function\": {\n",
    "                \"name\": \"retrieval_tool\",\n",
    "                \"description\": \"Retrieval tool for MedQA.\",\n",
    "            },\n",
    "            \"source_code\": inspect.getsource(retrieval_tool),\n",
    "        },\n",
    "        output=retrieved_data,\n",
    "        trace_parent_id=trace_id,\n",
    "        start_time=start_time,\n",
    "        end_time=datetime.now()\n",
    "    )\n",
    "    \n",
    "    # Populate the Prompt template\n",
    "    state_time=datetime.now()\n",
    "    messages = populate_template(template, inputs)\n",
    "    \n",
    "    # Call OpenAI to get response\n",
    "    chat_completion = openai.chat.completions.create(\n",
    "        model=model,\n",
    "        temperature=temperature,\n",
    "        messages=messages,\n",
    "    )\n",
    "\n",
    "    # Log the prompt information to Humanloop separately\n",
    "    humanloop.prompts.log(\n",
    "        path=\"evals_demo/medqa-answer\",\n",
    "        prompt={\n",
    "            \"model\": model,\n",
    "            \"temperature\": temperature,\n",
    "            \"template\": template,\n",
    "        },\n",
    "        inputs=inputs,\n",
    "        output=chat_completion.choices[0].message.content,\n",
    "        output_message=chat_completion.choices[0].message,\n",
    "        trace_parent_id=trace_id,\n",
    "        start_time=start_time,\n",
    "        end_time=datetime.now()\n",
    "    )\n",
    "    \n",
    "    return chat_completion.choices[0].message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1a69428-0dbe-489c-8664-339e31bc96aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update Flow logging to provide trace_id\n",
    "\n",
    "inputs = {\n",
    "    \"question\": \"A 34-year-old male suffers from inherited hemophilia A. He and his wife have three unaffected daughters. What is the probability that the second daughter is a carrier of the disease?\",\n",
    "    \"option_A\": \"0%\",\n",
    "    \"option_B\": \"25%\",\n",
    "    \"option_C\": \"50%\",\n",
    "    \"option_D\": \"75%\",\n",
    "    \"option_E\": \"100%\",\n",
    "}\n",
    "\n",
    "start_time = datetime.now()\n",
    "\n",
    "# Create the Flow log\n",
    "log = humanloop.flows.log(\n",
    "    path=\"evals_demo/medqa-flow\",\n",
    "    flow={\n",
    "        \"attributes\": {\n",
    "            \"prompt\": {\n",
    "                \"template\": template,\n",
    "                \"model\": model,\n",
    "                \"temperature\": temperature,\n",
    "            },\n",
    "            \"tool\": {\n",
    "                \"name\": \"retrieval_tool_v3\",\n",
    "                \"description\": \"Retrieval tool for MedQA.\",\n",
    "                \"source_code\": inspect.getsource(retrieval_tool),\n",
    "            },\n",
    "        }\n",
    "    },\n",
    "    inputs=inputs,\n",
    "    start_time=start_time,\n",
    ")\n",
    "\n",
    "chat_completion = ask_question(inputs, log.id)\n",
    "\n",
    "# Close the trace\n",
    "humanloop.flows.update_log(\n",
    "    log_id=log.id,\n",
    "    output=chat_completion.content,\n",
    "    trace_status=\"complete\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dfa2f59-0b80-48c7-b3b2-2ce7425b8ad7",
   "metadata": {},
   "source": [
    "![Flow Evals](../../assets/images/flow_full_trace.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20653258-0756-4aab-8308-2498e696cb2f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}