API key authentication for write replicas.
Service-to-service JWT authentication for write replicas.
Custom authentication headers for write replicas.
Configuration for a write replica endpoint.
agent_id and agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it reject
Run Schema with back-references for posting runs.
Middleware for propagating distributed tracing context using LangSmith.
This middleware checks for the 'langsmith-trace' header and propagates the
tracing context if present. It does not start new
String node extracted from the data.
Processes a list of string nodes for masking.
Configuration options for replacing sensitive data.
Declarative rule used for replacing sensitive data.
String node processor that uses a list of rules to replace sensitive data.
String node processor that uses a callable function to replace sensitive data.
Annotated type that will be stored as an attachment if used.
Protocol for binary IO-like objects.
Example base model.
Example upload with attachments.
Example create with attachments.
Info for an attachment.
Example model.
Operations to perform on attachments.
Example update with attachments.
Enum for dataset data types.
Dataset base model.
Schema for dataset transformations.
Dataset ORM model.
Class representing a dataset version.
Base Run schema.
A Run is a span representing a single unit of work or operation within your LLM app. This could be a single call to an LLM or chain, to a prompt formatting call, to a runnable lambda
Run schema when loading from the DB.
(Deprecated) Enum for run types. Use string directly.
Run-like dictionary, for type-hinting.
agent_id and agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it rejects the
Run schema with annotation queue info.
Base class for feedback sources.
This represents whether feedback is submitted from the API, model, human labeler, etc.
API feedback source.
Model feedback source.
Feedback source type.
Feedback schema.
agent_id / agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it rejects the feedback,
so it is
Specific value and label pair for feedback.
Represents how a feedback value ought to be interpreted.
Schema used for creating feedback.
Schema for getting feedback.
TracerSession schema for the API.
Sessions are also referred to as "Projects" in the UI.
A project, hydrated with additional information.
Sessions are also referred to as "Projects" in the UI.
A protocol representing objects similar to BaseMessage.
Represents the schema for a dataset share.
Represents a rubric item assigned to an annotation queue.
Links a feedback config to a queue with optional per-queue customization.
Represents an annotation queue.
Represents an annotation queue with details.
A run identified by its full lookup key, for adding to an annotation queue.
Unlike a bare run ID, this carries the partition key (session_id and
start_time) so the run can be located directly
Configuration for batch ingestion.
Information about the LangSmith server.
Settings for the LangSmith tenant.
Represents the schema for a feedback ingest token.
Run event schema.
Timedelta input schema.
Represents the difference information between two datasets.
Represents a comparative experiment.
This information summarizes evaluation results comparing two or more models on a given dataset.
Represents a Prompt with a manifest.
Represents a listed prompt commit with associated metadata.
Represents a Prompt with metadata.
A list of prompts with metadata.
Enum for sorting fields for prompts.
A file with inline content.
A link to another agent repo.
A link to a skill repo.
An agent pulled from hub.
A skill pulled from hub.
Commit details returned from a directory commit.
Response body for POST /directories/commits.
Breakdown of input token counts.
Does not need to sum to full input token count. Does not need to have all keys.
Breakdown of output token counts.
Does not need to sum to full output token count. Does not need to have all keys.
Breakdown of input token costs.
Does not need to sum to full input cost. Does not need to have all keys.
Breakdown of output token costs.
Does not need to sum to full output cost. Does not need to have all keys.
Usage metadata for a message, such as token counts.
This is a standard representation of token usage that is consistent across models.
Usage metadata dictionary extracted from a run.
Should be the same as UsageMetadata, but does not require all keys to be present.
Response object returned from the upsert_examples_multipart method.
Example with runs.
Run statistics for an experiment.
Results container for experiment data with stats and examples.
Breaking change in v0.4.32: The 'stats' field has been split into 'feedback_stats' and 'run_stats'.
An Insights Report created by the Insights Agent over a tracing project.
A trace highlighted in an insights report summary.
High-level summary of an insights job: key points and highlighted traces.
A single cluster of runs in an insights report.
Full result of fetching an Insights report (job + clusters + summary + optional runs).
A feedback key and weight used when calculating feedback formulas.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores vi
Schema used for creating a feedback formula.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores via the LangSmith UI ins
Schema used for updating a feedback formula.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores via the LangSmith UI ins
Schema for getting feedback formulas.
.. admonition:: Deprecated
Composite feedback formulas are no longer supported in the SDK.
Add composite feedback scores via the LangSmith UI instead. T
Represents a feedback configuration for a tenant's feedback key.
Feedback configurations define how feedback with a given key should be interpreted, including its type (continuous, categorical, or fr
An error occurred while communicating with the LangSmith API.
Internal server error while communicating with LangSmith.
Client took too long to send request body.
User error caused an exception when communicating with LangSmith.
You have exceeded the rate limit for the LangSmith API.
Couldn't authenticate with the LangSmith API.
Couldn't find the requested resource.
The resource already exists.
Couldn't connect to the LangSmith API.
Port of ExceptionGroup for Py < 3.11.
Base class for warnings.
Warning for missing API key.
Filter urllib3 warnings logged when the connection pool isn't reused.
Filter for retries from this lib.
Wrapper to filter logs with this name.
ThreadPoolExecutor that copies the context to the child thread.
A single cache entry with metadata for TTL tracking.
Cache performance metrics.
Thread-safe LRU cache with background thread refresh.
For use with the synchronous Client.
Features:
Thread-safe LRU cache with asyncio task refresh.
For use with the asynchronous AsyncClient.
Features:
Deprecated alias for PromptCache. Use PromptCache instead.
Deprecated alias for AsyncPromptCache. Use AsyncPromptCache instead.
Overrides for LangSmith runtime behavior.
This class allows overriding default async implementations for environments that don't support certain asyncio features (e.g., Temporal doesn't support ``run
A sentinel singleton class used to distinguish omitted keyword arguments from those passed in with the value None (which may have different behavior).
A class for making assertions on expectation values.
A class for setting expectations on test results.
Any additional info to be injected into the run dynamically.
Implementations of this Protocol accept an optional langsmith_extra parameter.
Manage a LangSmith run in context.
This class can be used as both a synchronous and asynchronous context manager.
agent_id / agent_environment are in beta. Agent a
A string that LangSmith serializes as [LANGSMITH SECRET].
Wrap a credential once, where it is read, and LangSmith masks it wherever it appears in a trace, at any nesting depth::
@traceable
Plugin for rendering LangSmith results.
Introduced in python 3.9.
Used for optional OTEL tracing.
Item returned by :meth:Client.list_threads.
Client for interacting with the LangSmith API.
One write destination: a server plus the credentials to reach it.
Async Client for interacting with the LangSmith API.
Optional per-mount cache configuration supported by bucket mounts.
Optional fields applied per bucket-backed sandbox mount.
Required S3 configuration for a sandbox mount.
S3 configuration for a sandbox mount.
S3-backed sandbox mount specification.
Required GCS configuration for a sandbox mount.
GCS configuration for a sandbox mount.
GCS-backed sandbox mount specification.
Git ref selected for a sandbox mount.
Required Git configuration for a sandbox mount.
Git configuration for a sandbox mount.
Git-backed sandbox mount specification.
Required Context Hub configuration for a sandbox mount.
Context Hub configuration for a sandbox mount.
Read-only Context Hub-backed sandbox mount specification.
AWS credentials used by the backend to authenticate S3 mounts.
IAM role restricted by the backend to this sandbox's S3 mount scopes.
GCP credentials used by the backend to authenticate GCS mounts.
Provider auth blocks for sandbox mounts.
Public mount config sent to the sandbox API.
What a command handle needs from its transport to steer a command.
A one-way transport supplies this too, and raises from the methods it cannot honor.
Async equivalent of :class:StreamControl.
Result of executing a command in a sandbox.
Lightweight provisioning status for any async-created resource.
The user, working directory and environment commands run with.
Mirrors docker run -u / -w / -e: user and work_dir replace the
layer below, env_vars merge into it key by key. It applie
One filesystem entry returned by :meth:Sandbox.glob.
Entries matching a glob pattern.
One matching line found by :meth:Sandbox.grep.
Lines matching a literal search.
What a HEAD on a sandbox file reports, without transferring it.
Bytes returned by a ranged read, and where they sit in the file.
Represents a sandbox snapshot.
Snapshots are built from Docker images or captured from running sandboxes. They are used to create new sandboxes.
One tag published under a snapshot name, and the snapshot it resolves to.
Authenticated URL for accessing an HTTP service running in a sandbox.
Properties auto-refresh the token transparently when it nears expiry.
HTTP helper methods (.get, .post, etc.) inject the
Async variant of :class:ServiceURL with auto-refreshing token.
Properties and HTTP helpers are async. Use with
:meth:AsyncSandboxClient.service or :meth:AsyncSandbox.service.
Example::
sv
A link that downloads one sandbox file with no LangSmith credential.
The link is pinned to the sandbox, the file path, and the response headers, so it cannot be repointed at another file. It is pinne
Service URL gated by LangSmith login rather than a token.
The grant is durable: there is no token to carry and no expiry, so the URL is only usable from a browser signed in to LangSmith. That is also
A single chunk of streaming output from command execution.
Handle to a running command with streaming output and auto-reconnect.
Iterable, yielding OutputChunk objects (stdout and stderr interleaved in arrival order). Access .result after iteration to get th
Async handle to a running command with streaming output and auto-reconnect.
Async iterable, yielding OutputChunk objects (stdout and stderr interleaved in arrival order). Access .result after iterati
Represents an active sandbox for running commands and file operations.
This class is typically obtained from SandboxClient.sandbox() and supports the context manager protocol for automatic cleanup.
TCP tunnel to a port inside a sandbox.
Opens a local TCP listener and forwards each accepted connection through a yamux-multiplexed WebSocket to the daemon, which dials the target port inside the san
Async wrapper around :class:Tunnel.
The underlying tunnel runs in background threads (TCP listener + bridges); async context-manager methods delegate to the sync tunnel via the event loop's executo
Raised when a sandbox user token or callback signature fails verification.
The LangSmith user a service URL request was made by.
The sandbox whose outbound request triggered a proxy callback.
Snapshot of the outbound request, sent for full_request callbacks.
A verified proxy callback payload.
Verifies tokens LangSmith signs for code running in or behind a sandbox.
Keys are fetched from LangSmith's JWKS endpoint and cached.
A single multiplexed stream within a yamux session.
Streams are created via :meth:YamuxSession.open_stream and provide
blocking read/write/close with per-stream flow control.
Client-side yamux session over a byte-stream connection.
The connection must implement read(n) -> bytes, write(data) -> int,
and close() -> None. Typically this is a :class:_WSAdapter
Represents an active sandbox for running commands and file operations async.
This class is typically obtained from AsyncSandboxClient.sandbox() and supports the async context manager protocol for aut
Sync httpx transport that retries on transient errors.
Retries on:
Async httpx transport that retries on transient errors.
Async equivalent of RetryTransport. See RetryTransport for details.
Async client for interacting with the Sandbox Server API.
This client provides an async interface for managing sandboxes and snapshots.
Client for interacting with the Sandbox Server API.
This client provides a simple interface for managing sandboxes and snapshots.
Base exception for sandbox client errors.
Raised when the API endpoint returns an unexpected error.
For example, this is raised for wrong URL or path.
Raised when authentication fails (invalid or missing API key).
Raised when connection to the sandbox server fails.
Raised when a transient failure occurs before a command can start.
run() retries this error with the same command ID, so the server can
deduplicate an attempt whose outcome is unknown.
Raised when the socket fails or times out before the WebSocket handshake.
The execute frame was never sent, so re-issuing the same command ID cannot double-run a command.
Raised when the server sends a 1001 Going Away close frame.
This indicates a server hot-reload, not a true connection failure. The command is still running on the server.
This is a subclass of Sandb
Raised when a resource is not found.
Raised when an operation times out.
Raised when deleting a resource that is still in use.
Raised when creating a resource that already exists.
Raised when updating a resource name to one that already exists.
Raised when request validation fails.
This includes:
Raised when organization quota limits are exceeded.
Users should contact technical support via our Support Portal (https://support.langchain.com) to increase quotas.
Raised when resource provisioning fails.
Raised when dataplane_url is not available for the sandbox.
This occurs when the sandbox-router URL is not configured for the cluster.
Raised when attempting to interact with a sandbox that is not ready.
Raised when a sandbox operation fails (run, read, write).
Raised when a command exceeds its timeout.
Base exception for TCP tunnel errors.
The daemon rejected the port as not allowed.
Nothing is listening on the target port inside the sandbox.
Protocol version mismatch between the tunnel client and the daemon.
A secret value that can be used by sandbox proxy rules.
Grant letting code inside a sandbox call the LangSmith API as the creator.
mode is "INHERIT" for everything the creator can do, or
"EXPLICIT" for the subset named in permissions. Perm
Represents the results of an evaluate() call.
This class provides an iterator interface to iterate over the experiment results as they become available. It also provides methods to access the experim
Represents the results of an evaluate_comparative() call.
This class provides an iterator interface to iterate over the comparison results, indexed access by example ID, and properties to access the
Configuration for a categorical score.
Configuration for a continuous score.
A class for building LLM-as-a-judge evaluators.
.. deprecated:: 0.5.0
LLMEvaluator is deprecated. Use openevals instead: https://github.com/langchain-ai/openevals
A category for categorical feedback.
Configuration to define a type of feedback.
Applied on on the first creation of a feedback_key.
Evaluation result.
Batch evaluation results.
This makes it easy for your evaluator to return multiple metrics at once.
Evaluator interface class.
Feedback scores for the results of comparative evaluations.
These are generated by functions that compare two or more runs, returning a ranking or other feedback.
A dynamic evaluator that wraps a function and transforms it into a RunEvaluator.
This class is designed to be used with the @run_evaluator decorator, allowing
functions that take a Run and an o
Compare predictions (as traces) from 2 or more runs.
Grades the run's string input, output, and optional answer.
.. deprecated:: 0.5.0
StringEvaluator is deprecated. Use openevals instead: https://github.com/langchain-ai/openevals
For parameters with a meaningful None value, we need to distinguish between the user explicitly passing None, and the user not passing the parameter at all.
User code shouldn't need to use not_given
To explicitly omit something from being sent in a request, use omit.
# as the default `Content-Type` header is `application/json` that will be sent
client.post("/upload/files", files={"file":
Represents a type that has inherited from Generic
The __orig_bases__ property can be used to determine the resolved
type variable for a given base class.
Used as a placeholder to easily convert runtime types to a Pydantic format to provide validation.
For example:
validated = RootModel[int](__root__="5").__root__
# validated: 5Provides the core interface to iterate over a synchronous stream response.
Provides the core interface to iterate over an asynchronous stream response.
Subclass of APIResponse providing helpers for dealing with binary data.
Note: If you want to stream the response data instead of eagerly reading it all at once then you should use `.with_streaming_re
Subclass of APIResponse providing helpers for dealing with binary data.
Note: If you want to stream the response data instead of eagerly reading it all at once then you should use `.with_streaming_re
Attempted to read or stream content, but the content has already been streamed.
This can happen if you use a method like .iter_lines() and then attempt
to read th entire response body afterwards, e
Context manager for ensuring that a request is not made until it is entered and that the response will always be closed when the context manager exits
Context manager for ensuring that a request is not made until it is entered and that the response will always be closed when the context manager exits
Raised when an API response has a status code of 4xx or 5xx.
Stores the necessary information to build the request to retrieve the next page.
Either url or params must be set.
Defines the core interface for pagination.
Implements data methods to pretend that an instance is another instance.
This includes forwarding attribute access and other methods.
A proxy for the langsmith._openapi_client.resources module.
This is used so that we can lazily import langsmith._openapi_client.resources only when
needed and so that users can just import `lan
Metadata class to be used in Annotated types to provide information about a given type.
For example:
class MyParams(TypedDict): account_holder_name: Annotated[str, PropertyInfo(alias='accountHol
Identity info for an assigned reviewer on an annotation queue.
AnnotationQueue schema with size.
Identity info for an assigned reviewer on an annotation queue.
AnnotationQueue schema.
Size of an Annotation Queue
completion_cost_details is the per-sub-category sum of completion cost details across the thread. Populated when COMPLETION_COST_DETAILS is selected.
completion_token_details is the per-sub-category sum of completion token details across the thread. Populated when COMPLETION_TOKEN_DETAILS is selected.
prompt_cost_details is the per-sub-category sum of prompt cost details across the thread. Populated when PROMPT_COST_DETAILS is selected.
prompt_token_details is the per-sub-category sum of prompt token details across the thread. Populated when PROMPT_TOKEN_DETAILS is selected.
Run schema with annotation queue info.
completion_cost_details is the per-category USD breakdown of completion_cost.
Categories mirror completion_token_details. Returned only when the COMPLETION_COST_DETAILS field is requested.
completion_token_details is the per-category breakdown of completion_tokens.
Category names are model-specific (for example reasoning, audio). Returned only when the `COMPLETION_TOKEN_DETAILS
prompt_cost_details is the per-category USD breakdown of prompt_cost.
Categories mirror prompt_token_details. Returned only when the PROMPT_COST_DETAILS field is requested.
prompt_token_details is the per-category breakdown of prompt_tokens.
Category names are model-specific (for example cache_read, cache_write). Returned only when the PROMPT_TOKEN_DETAILS fie
Identity info for an assigned reviewer on an annotation queue.
AnnotationQueue schema with rubric.
completion_cost_details is the USD cost breakdown for completion-side categories; per-category values are under raw. Omitted unless included in selects.
completion_token_details is the completion-side token breakdown by category; per-category counts are under raw. Omitted unless included in selects.
prompt_cost_details is the USD cost breakdown for prompt-side categories; per-category values are under raw. Omitted unless included in selects.
prompt_token_details is the prompt-side token breakdown by category; per-category counts are under nested raw. Omitted unless included in selects.
Add a single run to AQ (CH path) with an optional back-pointer to the issues-agent proposal that seeded this add. Use when bulk-adding runs that come from different proposals — each row carries its ow
Deprecated: use plain UUID list or AddRunToQueueByKeyRequest instead.
Add run to AQ by SmithDB key. is_root derived server-side (LSAQ-141).
sort controls feedback-score sorting (single project only).
Configure global LangSmith tracing context.
This function allows you to set global configuration options for LangSmith tracing that will be applied to all subsequent traced operations. It modifies co
Validate that the dict only contains allowed keys.
Create an anonymizer function.
Build an anonymizer pre-loaded with :data:DEFAULT_SECRET_RULES.
Pass the result to Client(anonymizer=...) to redact detected secrets
from run inputs, outputs, and metadata client-side, before u
Get the error message for an invalid prompt identifier.
Used consistently across the codebase when parsing prompt identifiers fails.
Return True if tracing is enabled.
Return True if testing is enabled.
Validate specified keyword args are mutually exclusive.
Raise an error with the response text.
Get the value of a string enum.
Log a message at the specified level, but only once.
Extract messages from the given inputs dictionary.
Retrieve the message generation from the given outputs.
Retrieve the prompt from the given inputs.
Get the LLM generation from the outputs.
Get the correct docker compose command for this system.
Convert a LangChain message to an example.
Check if the given object is similar to BaseMessage.
Check if the given environment variable is truish.
Retrieve an environment variable from a list of namespaces.
Get the project name for a LangSmith tracer.
Get the agent environment for a LangSmith tracer.
Experimental: in beta and enabled per workspace. A workspace without agent addressing rejects the runs, so tracing is lost rather than falling back t
Get the agent ID for a LangSmith tracer.
Experimental: in beta and enabled per workspace. A workspace without agent addressing rejects the runs, so tracing is lost rather than falling back to a proje
Temporarily adds specified filters to a logger.
Parameters:
logging.Filter objects to be temporarily added
to tGet the testing cache directory.
Filter request headers based on ignore_hosts and allow_hosts.
Use a cache for requests.
Use a cache for requests.
Deep copy a value with a compromise for uncopyable objects.
Check if the current version is greater or equal to the target version.
Parse a string in the format of owner/name:hash, name:hash, owner/name, or name.
Parse a hub repo identifier (owner/name:hash, name, etc.).
Agents, skills, and prompts share the same identifier grammar on Hub.
Get the LangSmith API URL from the environment or the given value.
Get the API key from the environment or the given value.
Get workspace ID.
Get the host URL based on the web URL or API URL.
Check if the value is truish.
Configure the global prompt cache.
This should be called before any cache instances are created or used.
Configure the global prompt cache.
This should be called before any cache instances are created or used.
Set LangSmith runtime overrides.
This allows customizing LangSmith's async runtime behavior for environments
with constrained async runtimes (e.g., Temporal, which doesn't support
run_in_executor
Get the current runtime overrides.
Get the current run tree.
Uses a weakref-based lookup to avoid memory leaks from captured contexts. The RunTree may return None if it has been garbage collected.
Set a RunTree as the active tracing parent within this block.
Unlike tracing_context, this only sets the parent run tree and nothing
else, making it safe to use in isolated threads where you want p
Update metadata on the current run tree.
Get the current tracing context.
Set the tracing context for a block of code.
agent_id / agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it rejects
Check if a function is @traceable decorated.
Ensure that a function is traceable.
Inspect function or wrapped function to see if it is async.
Trace a function with langsmith.
agent_id / agent_environment are in beta. Agent addressing is
enabled per workspace; a workspace without it rejects the runs, s
Convert a function wrapped by the LangSmith @traceable decorator to a Runnable.
Set a boolean flag for LangSmith output.
Skip if --langsmith-output is already defined.
Call immediately after command line options are parsed (pytest v7).
Handle args in pytest v8+.
Apply LangSmith tracking to tests marked with @pytest.mark.langsmith.
Remove the short test-status character outputs ("./F").
Register the 'langsmith' marker.
Close the session.
Convert a prompt to OpenAI format.
Requires the langchain_openai package to be installed.
Convert a prompt to Anthropic format.
Requires the langchain_anthropic package to be installed.
Dump model depending on pydantic version.
Format the object so its Prompt Hub compatible.
Generate a random UUID v7.
Generate a UUID v7 from a datetime.
Compute the run ID used for a secondary tracing replica.
Whether name is on, per LANGSMITH_FEATURES and its default.
Render the env var assignment that turns name on or off.
Get information about the git repository.
Get the runtime information as well as metrics.
Get CPU and other performance metrics.
Get information about the environment.
Get information about the environment.
Retrieve the langchain environment variables.
Retrieve the langchain environment variables.
Build an S3-backed sandbox mount specification.
Build a public Git-backed sandbox mount specification.
Build a GCS-backed sandbox mount specification.
Build a read-only Context Hub-backed sandbox mount specification.
The repo's latest commit tree is mirrored into mount_path and kept in
sync for the sandbox's lifetime unless ``initial_pull_only`
Build a high-level mount config from provider auth and mount specs.
The returned value is sent as the public mount_config field. The
backend expands provider auth into runtime proxy rules.
For S
Reject explicit provider proxy rules owned by mount_config auth.
Merge request headers, giving precedence to overrides.
Names are normalized to lowercase so an override replaces a base header that differs only in casing. HTTP header names are case-insensitive, so
Normalize a per-command run_config, rejecting the deprecated pairing.
The server answers 400 for a request carrying both spellings rather than letting one silently win, so refuse it here where the me
Whether to half-close stdin at spawn.
Defaults on for a non-PTY command: a command that reads stdin otherwise blocks on a pipe nobody writes to until the timeout kills it. A PTY has no separate write
Render a byte range as an RFC 9110 Range header value.
Build a FileStat from a HEAD response's headers.
Build a FileChunk from a ranged download response.
Map a file-operation HTTP error, including the non-JSON 416.
Validate parameters for service URL generation.
Validate a TTL value for sandbox create/update.
Parse standardized error response.
Expected format: {"detail": {"error": "...", "message": "..."}}
Returns a dict with:
Parse error response (simplified version for sandbox operations).
Returns a dict with:
Parse Pydantic validation error response.
Returns a list of validation error details, each containing:
Extract quota type from error message.
Returns one of: "sandbox_count", "cpu", "memory", "storage", or None.
Raise ResourceCreationError with the error_type from the API response.
The error_type indicates the specific failure reason:
Handle HTTP errors specific to sandbox creation.
Maps API error responses to specific exception types:
Raise SandboxNotReadyError when the API rejected a proxy-config update.
A proxy config can only be written to a ready sandbox, and that status
check is the sole source of InvalidRequest o
Handle HTTP errors and raise appropriate exceptions (for client operations).
Handle HTTP errors for sandbox operations (run, read, write).
Maps API error types to specific exceptions:
Whether the SSE exec feature is on.
Reject run() options that need the WebSocket's control channel.
Per-event read timeout, in seconds. <= 0 disables it.
Build the body for /execute/stream/start.
Start a command over SSE, yielding WebSocket-shaped message dicts.
Returns (messages, control). The iterator transparently re-requests the
stream whenever the server asks for an ack or the connec
Reattach to a command started earlier, resuming from the given offsets.
Async equivalent of :func:run_sse_stream.
Async equivalent of :func:resume_sse_stream.
Monotonic instant after which connect attempts must stop, if bounded.
Per-attempt open timeout, clamped so it cannot outlive deadline.
Execute a command over WebSocket, yielding raw message dicts.
Returns a tuple of (message_iterator, control). The control object provides send_kill() and send_input() methods for the CommandHandle.
Reconnect to an existing command over WebSocket.
Returns a tuple of (message_iterator, control), same as run_ws_stream.
The iterator yields the server's started acknowledgement, then stdout,
stde
Async equivalent of run_ws_stream.
Returns (async_message_iterator, async_control).
Async equivalent of reconnect_ws_stream.
Create a LangSmith workspace secret reference for a proxy configuration.
Provide a write-only secret value for a proxy configuration.
The value is sent when creating or updating the sandbox proxy config, but LangSmith stores it as an opaque secret and does not return it f
Build a sandbox proxy config from one or more proxy rules.
Use provider-specific rule helpers such as aws_auth and gcp_auth
when a sandbox needs multiple auth flows.
Build a sandbox proxy rule that signs AWS HTTPS requests.
The sandbox proxy keeps the real AWS credentials outside the sandbox and signs supported AWS requests with SigV4 on the sandbox's behalf. Pro
Build a sandbox proxy rule that injects GCP OAuth bearer auth.
The sandbox proxy keeps the service account JSON outside the sandbox and injects OAuth bearer tokens for built-in Google API host matchi
Patch the Anthropic client to make it traceable.
Patch the Google Gen AI client to make it traceable.
BETA: This wrapper is in beta.
Patch the OpenAI client to make it traceable.
Evaluate a target system on a given dataset.
Evaluate existing experiment runs.
Evaluate existing experiment runs against each other.
This lets you use pairwise preference scoring to generate more reliable feedback in your experiments.
Evaluate an async target system on a given dataset.
Evaluate existing experiment runs asynchronously.
Chain multiple async iterables.
Convert a list of examples to an async iterable.
Create a run evaluator from a function.
Decorator that transforms a function into a RunEvaluator.
Create a comaprison evaluator from a function.
Generate a random name.
Copy only the containers along the given paths.
Used to guard against mutation by extract_files without copying the entire structure. Only dicts and lists that lie on a path are copied; everything el
Returns whether or not the given type is either a BaseModel or a union of BaseModel
Construct a BaseModel class without validation.
This is useful for cases where you need to instantiate a BaseModel
from an API response as this provides type-safe params which isn't supported
by he
Loose coercion to the expected type with construction of nested values.
Note: the returned value from this function is not guaranteed to match the given type.
Loose coercion to the expected type with construction of nested values.
If the given value does not match the expected type then it is returned as-is.
Strict validation that the given value matches the expected type
Add a pydantic config for the given type.
Note: this is a no-op on Pydantic v1.
TypeGuard for determining whether or not the given type is a subclass of Stream / AsyncStream
Given a type like Stream[T], returns the generic type variable T.
This also handles the case where a concrete subclass is given, e.g.
class MyStream(Stream[bytes]):
...
extract_stream_
Higher order function that takes one of our bound API methods and wraps it
to support streaming and returning the raw APIResponse object directly.
Higher order function that takes one of our bound API methods and wraps it
to support streaming and returning the raw APIResponse object directly.
Higher order function that takes one of our bound API methods and an APIResponse class
and wraps the method to support streaming and returning the given response class directly.
Note: the given `re
Higher order function that takes one of our bound API methods and an APIResponse class
and wraps the method to support streaming and returning the given response class directly.
Note: the given `re
Higher order function that takes one of our bound API methods and wraps it
to support returning the raw APIResponse object directly.
Higher order function that takes one of our bound API methods and wraps it
to support returning the raw APIResponse object directly.
Higher order function that takes one of our bound API methods and an APIResponse class
and wraps the method to support returning the given response class directly.
Note: the given response_cls *m
Higher order function that takes one of our bound API methods and an APIResponse class
and wraps the method to support returning the given response class directly.
Note: the given response_cls *m
Given a type like APIResponse[T], returns the generic type variable T.
This also handles the case where a concrete subclass is given, e.g.
class MyResponse(APIResponse[bytes]):
...
ext
Create a dict of type RequestOptions without keys of NotGiven values.
Interpolate {name} placeholders in template from keyword arguments.
Take a blocking function and create an async one that receives the same positional and keyword arguments.
Usage:
def blocking_func(arg1, arg2, kwarg1=None):
# blocking code
return
If the given type is typing.Iterable[T]
Return whether the provided argument is an instance of TypeAliasType.
type Int = int
is_type_alias_type(Int)
# > True
Str = TypeAliasType("Str", str)
is_type_alias_type(Str)
# > TrueGiven a type like Foo[T], returns the generic type variable T.
This also handles the case where a concrete subclass is given, e.g.
class MyResponse(Foo[bytes]):
...
extract_type_var(My
Serialize an object to UTF-8 encoded JSON bytes.
Extends the standard json.dumps with support for additional types
commonly used in the SDK, such as datetime, pydantic.BaseModel, etc.
Returns whether or not the given function has a specific parameter
Ensure that the signature of the second function matches the first.
Wrapper over transform() that allows None to be passed.
See transform() for more details.
Transform dictionaries based off of type information from the given type, for example:
class Params(TypedDict, total=False):
card_id: Required[Annotated[str, PropertyInfo(alias="cardID")]]
Wrapper over async_transform() that allows None to be passed.
See async_transform() for more details.
Transform dictionaries based off of type information from the given type, for example:
class Params(TypedDict, total=False):
card_id: Required[Annotated[str, PropertyInfo(alias="cardID")]]
Recursively extract files from the given dictionary based on specified paths.
A path may look like this ['foo', 'files', '
array_format controls how <array> segments contri
Add single quotation marks around the given string. Does not do any escaping.
Decorator to enforce a given set of arguments or variants of arguments are passed to the decorated function.
Useful for enforcing runtime validation of overloaded functions.
Example usage:
@ov
Remove all top-level keys where their values are instances of NotGiven
Remove a prefix from a string.
Backport of str.removeprefix for Python < 3.9
Remove a suffix from a string.
Backport of str.removesuffix for Python < 3.9
A version of functools.lru_cache that retains the type signature for the wrapped function arguments.
Translates a mapping / sequence recursively in the same fashion
as pydantic v2's model_dump(mode="json").
Parse a datetime/int/float/string and return a datetime.datetime.
This function supports time zone offsets. When the input contains one, the output uses a timezone with a fixed offset from UTC.
Rais
Parse a date/int/float/string and return a datetime.date.
Raise ValueError if the input is well formatted but not a valid date. Raise ValueError if the input isn't well formatted.
LangSmith Client.
Schemas for the LangSmith API.
Middleware for making it easier to do distributed tracing.
Schemas for the LangSmith API.
Generic utility functions.
Prompt caching module for LangSmith SDK.
This module provides thread-safe LRU caches with background refresh for prompt caching. Includes both sync and async implementations.
Make approximate assertions as "expectations" on test results.
This module is designed to be used within test cases decorated with the
@pytest.mark.decorator decorator
It allows you to log scores
Decorator for creating a run tree from functions.
Mark a value as a secret, so LangSmith masks it instead of tracing it.
::
from langsmith import LangSmithSecret
API_KEY = LangSmithSecret(os.environ["VENDOR_API_KEY"])
See :class:`LangSmit
LangSmith Pytest hooks.
Client for interacting with the LangSmith API.
Use the client to customize API keys / workspace connections, SSL certs, etc. for tracing.
Also used to create, read, update, and delete LangSmith reso
Public UUID v7 helpers.
These helpers expose utilities for generating UUID v7 identifiers in user code.
The Async LangSmith Client.
Utilities to get information about the runtime environment.
LangSmith Sandbox Module.
This module provides sandboxed code execution capabilities through the LangSmith Sandbox API.
This module provides convenient tracing wrappers for popular libraries.
Evaluation Helpers.
Contains the LLMEvaluator class for building LLM-as-a-judge evaluators.
This module contains the evaluator classes for evaluating runs.
This module contains the StringEvaluator class.
LangSmith pytest testing module.
This package contains the Python client for interacting with the LangSmith platform.
To install:
pip install -U langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=ls_...Then trace:
import openai
from langsmith.wrappers import wrap_openai
from langsmith import traceable
# Auto-trace LLM calls in-context
client = wrap_openai(openai.Client())
@traceable # Auto-trace this function
def pipeline(user_input: str):
result = client.chat.completions.create(
messages=[{"role": "user", "content": user_input}],
model="gpt-5.4"
)
return result.choices[0].message.content
pipeline("Hello, world!")Every LLM call inside pipeline is nested under a single trace in the LangSmith UI.
LangSmith helps you and your team develop and evaluate language models and intelligent agents. It is compatible with any LLM application.
Cookbook: For tutorials on how to get more value out of LangSmith, check out the Langsmith Cookbook repo.
A typical workflow looks like:
We'll walk through these steps in more detail below.
When sandbox code needs to call AWS services, use the sandbox AWS auth proxy. The proxy keeps the real AWS credentials outside the sandbox and signs supported AWS HTTPS requests with SigV4, so code in the sandbox can use AWS SDKs normally without storing long-lived AWS keys in files, environment variables, shell history, or logs.
Store AWS credentials as LangSmith workspace secrets using names that make sense for your workspace. Then create the sandbox with an AWS auth proxy config:
from langsmith.sandbox import (
SandboxClient,
aws_auth,
proxy_config,
workspace_secret,
)
client = SandboxClient()
auth_config = proxy_config(
rules=[
aws_auth(
access_key_id=workspace_secret("SANDBOX_AWS_ACCESS_KEY_ID"),
secret_access_key=workspace_secret("SANDBOX_AWS_SECRET_ACCESS_KEY"),
)
],
)
with client.sandbox(
name="aws-sandbox",
proxy_config=auth_config,
) as sandbox:
result = sandbox.run("python your_aws_script.py")
print(result.stdout)Use opaque_secret("...") instead of workspace_secret(...) when your
application needs to pass short-lived write-only AWS credentials at sandbox
creation time. Plaintext AWS credential values are not accepted directly; wrap
them as opaque_secret(...) values.
When sandbox code needs to call Google APIs, use the sandbox GCP auth proxy. The proxy keeps the service account JSON outside the sandbox and injects OAuth bearer tokens for Google API hosts matched automatically by the sandbox proxy.
Store the service account JSON as a LangSmith workspace secret. Then create the sandbox with a GCP auth proxy config:
from langsmith.sandbox import (
SandboxClient,
gcp_auth,
proxy_config,
workspace_secret,
)
client = SandboxClient()
auth_config = proxy_config(
rules=[
gcp_auth(
service_account_json=workspace_secret(
"SANDBOX_GCP_SERVICE_ACCOUNT_JSON"
),
scopes=["https://www.googleapis.com/auth/devstorage.read_write"],
)
],
)
with client.sandbox(
name="gcp-sandbox",
proxy_config=auth_config,
) as sandbox:
result = sandbox.run("python your_gcp_script.py")
print(result.stdout)Use opaque_secret("...") for short-lived write-only service account JSON.
Plaintext service account JSON is not accepted directly.
When you create a LangSmith sandbox that needs filesystem access to external
data such as object storage buckets or public Git repositories, pass a
mount_config on sandbox creation. Mount specs contain only the mount target.
Provider credentials stay in mount_config.auth; the backend expands them into
runtime proxy auth rules. You can also pass proxy_config for non-mount proxy
behavior such as custom headers, callbacks, access control, and generic egress
rules. Explicit AWS/GCP proxy auth rules conflict with mount_config auth for
the same provider.
S3 mounts require AWS auth:
from langsmith.sandbox import (
aws_auth,
mount_config,
s3_mount,
workspace_secret,
)
mount_cfg = mount_config(
auth=[
aws_auth(
access_key_id=workspace_secret("SANDBOX_AWS_ACCESS_KEY_ID"),
secret_access_key=workspace_secret("SANDBOX_AWS_SECRET_ACCESS_KEY"),
)
],
mounts=[
s3_mount(
id="customer_data",
mount_path="/mnt/mounts/customer-data",
bucket="example-bucket",
prefix="datasets/customer-data",
region="us-east-1",
endpoint_url="https://s3.amazonaws.com",
path_style=False,
read_only=False,
)
],
)
with client.sandbox(
name="s3-mount-sandbox",
mount_config=mount_cfg,
) as sandbox:
result = sandbox.run("ls /mnt/mounts/customer-data")
print(result.stdout)GCS mounts require GCP auth:
from langsmith.sandbox import (
gcp_auth,
gcs_mount,
mount_config,
workspace_secret,
)
mount_cfg = mount_config(
auth=[
gcp_auth(
service_account_json=workspace_secret(
"SANDBOX_GCP_SERVICE_ACCOUNT_JSON"
)
)
],
mounts=[
gcs_mount(
id="customer_data",
mount_path="/mnt/mounts/customer-data",
bucket="example-bucket",
prefix="datasets/customer-data",
)
],
)
with client.sandbox(
name="gcs-mount-sandbox",
mount_config=mount_cfg,
) as sandbox:
result = sandbox.run("ls /mnt/mounts/customer-data")
print(result.stdout)Public Git mounts do not require AWS or GCP auth:
from langsmith.sandbox import git_mount, mount_config
mount_cfg = mount_config(
mounts=[
git_mount(
id="repo",
mount_path="/mnt/repo",
remote_url="https://github.com/langchain-ai/langsmith-sdk.git",
ref={"type": "branch", "name": "main"},
refresh_interval_seconds=60,
)
],
)
with client.sandbox(
name="git-mount-sandbox",
mount_config=mount_cfg,
) as sandbox:
result = sandbox.run("ls /mnt/repo")
print(result.stdout)Private Git repositories can use low-level proxy_config rules when the remote
requires proxy-managed auth. There is not yet a high-level private Git auth
helper.
Sign up for LangSmith using your GitHub, Discord accounts, or an email address and password. If you sign up with an email, make sure to verify your email address before logging in.
Then, create a unique API key on the Settings Page, which is found in the menu at the top right corner of the page.
[!NOTE] Save the API Key in a secure location. It will not be shown again.
You can log traces natively using the LangSmith SDK or within your LangChain application.
LangSmith seamlessly integrates with the Python LangChain library to record traces from your LLM applications.
Tracing can be activated by setting the following environment variables or by manually specifying the LangChainTracer.
import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_ENDPOINT"] = "https://api.smith.langchain.com"
# os.environ["LANGSMITH_ENDPOINT"] = "https://eu.api.smith.langchain.com" # If signed up in the EU region
os.environ["LANGSMITH_API_KEY"] = "<YOUR-LANGSMITH-API-KEY>"
# os.environ["LANGSMITH_PROJECT"] = "My Project Name" # Optional: "default" is used if not set
# os.environ["LANGSMITH_WORKSPACE_ID"] = "<YOUR-WORKSPACE-ID>" # Required for org-scoped API keysTip: Projects are groups of traces. All runs are logged to a project. If not specified, the project is set to
default.
If the environment variables are correctly set, your application will automatically connect to the LangSmith platform.
from langchain_core.runnables import chain
@chain
def add_val(x: dict) -> dict:
return {"val": x["val"] + 1}
add_val({"val": 1})You can still use the LangSmith development platform without depending on any LangChain code.
import os
os.environ["LANGSMITH_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGSMITH_API_KEY"] = "<YOUR-LANGSMITH-API-KEY>"
# os.environ["LANGSMITH_PROJECT"] = "My Project Name" # Optional: "default" is used if not setThe easiest way to log traces using the SDK is via the @traceable decorator. Below is an example.
from datetime import datetime
from typing import List, Optional, Tuple
import openai
from langsmith import traceable
from langsmith.wrappers import wrap_openai
client = wrap_openai(openai.Client())
@traceable
def argument_generator(query: str, additional_description: str = "") -> str:
return client.chat.completions.create(
model="gpt-5.4",
messages=[
{"role": "system", "content": "You are a debater making an argument on a topic."
f"{additional_description}"
f" The current time is {datetime.now()}"},
{"role": "user", "content": f"The discussion topic is {query}"}
]
).choices[0].message.content
@traceable
def argument_chain(query: str, additional_description: str = "") -> str:
argument = argument_generator(query, additional_description)
# ... Do other processing or call other functions...
return argument
argument_chain("Why is blue better than orange?")Alternatively, you can manually log events using the Client directly or using a RunTree, which is what the traceable decorator is meant to manage for you!
A RunTree tracks your application. Each RunTree object is required to have a name and run_type. These and other important attributes are as follows:
name: str - used to identify the component's purposerun_type: str - Currently one of "llm", "chain" or "tool"; more options will be added in the futureinputs: dict - the inputs to the componentoutputs: Optional[dict] - the (optional) returned values from the componenterror: Optional[str] - Any error messages that may have arisen during the callfrom langsmith.run_trees import RunTree
parent_run = RunTree(
name="My Chat Bot",
run_type="chain",
inputs={"text": "Summarize this morning's meetings."},
# project_name= "Defaults to the LANGSMITH_PROJECT env var"
)
parent_run.post()
# .. My Chat Bot calls an LLM
child_llm_run = parent_run.create_child(
name="My Proprietary LLM",
run_type="llm",
inputs={
"prompts": [
"You are an AI Assistant. The time is XYZ."
" Summarize this morning's meetings."
]
},
)
child_llm_run.post()
child_llm_run.end(
outputs={
"generations": [
"I should use the transcript_loader tool"
" to fetch meeting_transcripts from XYZ"
]
}
)
child_llm_run.patch()
# .. My Chat Bot takes the LLM output and calls
# a tool / function for fetching transcripts ..
child_tool_run = parent_run.create_child(
name="transcript_loader",
run_type="tool",
inputs={"date": "XYZ", "content_type": "meeting_transcripts"},
)
child_tool_run.post()
# The tool returns meeting notes to the chat bot
child_tool_run.end(outputs={"meetings": ["Meeting1 notes.."]})
child_tool_run.patch()
child_chain_run = parent_run.create_child(
name="Unreliable Component",
run_type="tool",
inputs={"input": "Summarize these notes..."},
)
child_chain_run.post()
try:
# .... the component does work
raise ValueError("Something went wrong")
child_chain_run.end(outputs={"output": "foo"})
child_chain_run.patch()
except Exception as e:
child_chain_run.end(error=f"I errored again {e}")
child_chain_run.patch()
pass
# .. The chat agent recovers
parent_run.end(outputs={"output": ["The meeting notes are as follows:..."]})
res = parent_run.patch()
res.result()Once your runs are stored in LangSmith, you can convert them into a dataset. For this example, we will do so using the Client, but you can also do this using the web interface, as explained in the LangSmith docs.
from langsmith import Client
client = Client()
dataset_name = "Example Dataset"
# We will only use examples from the top level AgentExecutor run here,
# and exclude runs that errored.
runs = client.list_runs(
project_name="my_project",
execution_order=1,
error=False,
)
dataset = client.create_dataset(dataset_name, description="An example dataset")
for run in runs:
client.create_example(
inputs=run.inputs,
outputs=run.outputs,
dataset_id=dataset.id,
)Check out the LangSmith Testing & Evaluation dos for up-to-date workflows.
For generating automated feedback on individual runs, you can run evaluations directly using the LangSmith client.
from typing import Optional
from langsmith.evaluation import StringEvaluator
def jaccard_chars(output: str, answer: str) -> float:
"""Naive Jaccard similarity between two strings."""
prediction_chars = set(output.strip().lower())
answer_chars = set(answer.strip().lower())
intersection = prediction_chars.intersection(answer_chars)
union = prediction_chars.union(answer_chars)
return len(intersection) / len(union)
def grader(run_input: str, run_output: str, answer: Optional[str]) -> dict:
"""Compute the score and/or label for this run."""
if answer is None:
value = "AMBIGUOUS"
score = 0.5
else:
score = jaccard_chars(run_output, answer)
value = "CORRECT" if score > 0.9 else "INCORRECT"
return dict(score=score, value=value)
evaluator = StringEvaluator(evaluation_name="Jaccard", grading_function=grader)
runs = client.list_runs(
project_name="my_project",
execution_order=1,
error=False,
)
for run in runs:
client.evaluate_run(run, evaluator)LangSmith easily integrates with your favorite LLM framework.
We provide a convenient wrapper for the OpenAI SDK.
In order to use, you first need to set your LangSmith API key.
export LANGSMITH_API_KEY=<your-api-key>Next, you will need to install the LangSmith SDK:
pip install -U langsmithAfter that, you can wrap the OpenAI client:
from openai import OpenAI
from langsmith import wrappers
client = wrappers.wrap_openai(OpenAI())Now, you can use the OpenAI client as you normally would, but now everything is logged to LangSmith!
client.chat.completions.create(
model="gpt-5.4",
messages=[{"role": "user", "content": "Say this is a test"}],
)Oftentimes, you use the OpenAI client inside of other functions.
You can get nested traces by using this wrapped client and decorating those functions with @traceable.
See this documentation for more documentation how to use this decorator
from langsmith import traceable
@traceable(name="Call OpenAI")
def my_function(text: str):
return client.chat.completions.create(
model="gpt-5.4",
messages=[{"role": "user", "content": f"Say {text}"}],
)
my_function("hello world")We provide a convenient integration with Instructor, largely by virtue of it essentially just using the OpenAI SDK.
In order to use, you first need to set your LangSmith API key.
export LANGSMITH_API_KEY=<your-api-key>Next, you will need to install the LangSmith SDK:
pip install -U langsmithAfter that, you can wrap the OpenAI client:
from openai import OpenAI
from langsmith import wrappers
client = wrappers.wrap_openai(OpenAI())After this, you can patch the OpenAI client using instructor:
import instructor
client = instructor.patch(OpenAI())Now, you can use instructor as you normally would, but now everything is logged to LangSmith!
from pydantic import BaseModel
class UserDetail(BaseModel):
name: str
age: int
user = client.chat.completions.create(
model="gpt-5.4",
response_model=UserDetail,
messages=[
{"role": "user", "content": "Extract Jason is 25 years old"},
]
)Oftentimes, you use instructor inside of other functions.
You can get nested traces by using this wrapped client and decorating those functions with @traceable.
See this documentation for more documentation how to use this decorator
@traceable()
def my_function(text: str) -> UserDetail:
return client.chat.completions.create(
model="gpt-5.4",
response_model=UserDetail,
messages=[
{"role": "user", "content": f"Extract {text}"},
]
)
my_function("Jason is 25 years old")The LangSmith pytest plugin lets Python developers define their datasets and evaluations as pytest test cases. See online docs for more information.
This plugin is installed as part of the LangSmith SDK, and is enabled by default. See also official pytest docs: How to install and use plugins
To learn more about the LangSmith platform, check out the docs.
The LangSmith SDK is licensed under the MIT License.
The copyright information for certain dependencies' are reproduced in their corresponding COPYRIGHT.txt files in this repo, including the following:
