LangSmith

LangSmith#

Langsmith is a service that monitors and evaluates your models. It is integrated with LangChain and LangGraph, but you can generally use it with any framework you like.

Note. Langsmith is closed-source proprietary roject. It provides some free to use features.

Check example for tracing exatly LangChain.

Setup#

Before importing langchain you’re supposed to set variables:

  • LANGSMITH_TRACING=true.

  • LANGSMITH_API_KEY=<your API key>.

If your API key has access to multiple workspaces, set LANGSMITH_WORKSPACE_ID.

Define the porject that is suppposed to be used with variable LANGSMITH_PROJECT.

Run the following cell:

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Please respond to the user's request only based on the given context."),
    ("user", "Question: {question}\nContext: {context}")
])

model = ChatOllama(model="qwen2:0.5b")
output_parser = StrOutputParser()
chain = prompt | model | output_parser

question = "What is the last name of Vlad"
context = "Vlad Smith, Andrew Gather"

ans = chain.invoke({"question": question, "context": context})
print(ans)

The last name of Vlad is Smith.

A new run associated with this code should appear in your “default” project.

Traceble#

You can trace to the langsmith calls of an arbitrary function by wrapping it in the @traceble decorator.

The following cell defines the some_super_function wrapped with @traceble.

from langsmith import traceable

@traceable
def some_super_function(a: str, b: str) -> dict[str, str | dict]:
    return {
        b: a,
        "inner info": {"this": "is information"}
    }

Running the following cell generates a message in Langsmith that contains the input and output values of the call.

some_super_function(20, 40)

{40: 20, 'inner info': {'this': 'is information'}}

Datasets#

LangSmith allows you to define datasets. These datasets can be created and edited from the LangSmith GUI or from python code.

The following code demonstrates how to create a dataset and push some samples to it.

from langsmith import Client

client = Client()

examples = [
    {
        "inputs": {"question": "Hello, how are you?"},
        "outputs": {"answer": "I'm fine and you."},
    },
    {
        "inputs": {"question": "What is the capital of France?"},
        "outputs": {"answer": "Paris is the capital of France"},
    },
    {
        "inputs": {"question": "Hi my name is Fedor"},
        "outputs": {"answer": "I'm some usefull LLM"},
    },
]

dataset_name = "Example dataset"
dataset = client.create_dataset(dataset_name=dataset_name)
client.create_examples(
    dataset_id=dataset.id,
    examples=examples
)

{'example_ids': ['678d0c77-7dca-4d3c-8485-493bfbd16c7b',
  'd478a242-b452-4cff-a84a-bb709f7b9cfc',
  '10b0f183-9dcf-499a-a4ee-9d427c64c460'],
 'count': 3}
Contents