Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# RubyLLM::Test

This gem provides testing utilities for RubyLLM, a Ruby library for working with large language models (LLMs). It enables calls to LLM's to be stubbed so that the surrounding application logic can be tested without making actual calls to the LLM. This is particularly useful for testing code that interacts with LLMs, as it allows developers to simulate responses from the LLM without incurring the cost, latency, or randomness of real API calls.
This gem provides testing utilities for RubyLLM, a Ruby library for working with large language models (LLMs). It enables calls to LLMs to be stubbed so that the surrounding application logic can be tested without making actual calls to the LLM. This is particularly useful for testing code that interacts with LLMs, as it allows developers to simulate responses from the LLM without incurring the cost, latency, or randomness of real API calls.

```ruby
RubyLLM::Test.stub_response("Outlook good")
Expand All @@ -13,7 +13,7 @@ assert_equal "Outlook good", response.content

## Installation

Add this line to your application's Gemfile:
Add this line to your application's Gemfile in the test group:

```ruby
gem 'ruby_llm-test'
Expand Down Expand Up @@ -109,6 +109,23 @@ RubyLLM::Test.with_responses('Hello, world!') do
end
```

### Testing Arguments

You can verify arguments passed to the LLM by checking the requests received by the test provider with methods `requests` and `last_request`.

```ruby
RubyLLM::Test.stub_response('Hello, world!')
chat = RubyLLM.chat(model: 'gpt-5-nano')
chat.with_tools(GreeterTool)
chat.ask('Hello?')
request = RubyLLM::Test.last_request

assert_equal 'gpt-5-nano', request.model
assert_includes request.tool_classes, GreeterTool
```

Any parameters passed to the Provider's `complete` method are available on the request object. The `tool_classes` method is a helper that returns the classes of any tools included in the request.

## Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
Expand Down
4 changes: 4 additions & 0 deletions lib/ruby_llm/test/complete_parameters.rb
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ def respond_to_missing?(name, include_private = false)
key?(name) || super
end

def tool_classes
(kwargs[:tools] || {}).values.map(&:class)
end

private

def positional_name_to_value
Expand Down
47 changes: 47 additions & 0 deletions test/system/tools_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# frozen_string_literal: true

require "test_helper"

class ToolsTest < Minitest::Test
class ToolOne < RubyLLM::Tool
description "This is the first tool"
end

class ToolTwo < RubyLLM::Tool
description "This is the second tool, it has a constructor"

def initialize(foo:)
super()
@foo = foo
end
end

def setup
RubyLLM::Test.reset
RubyLLM::Test.stub_response("stubbed response")
end

def test_tools_can_be_inspected
tool_two_instance = ToolTwo.new(foo: "bar")

chat = RubyLLM::Chat.new(model: "gpt-4-turbo")
chat.with_tools(ToolOne, tool_two_instance)
chat.ask("What is the meaning of life?")

tools = RubyLLM::Test.last_request.tools

assert_kind_of ToolOne, tools[ToolOne.new.name.to_sym]
assert_equal tool_two_instance, tools[tool_two_instance.name.to_sym]
end

def test_tool_classes_can_be_inspected
chat = RubyLLM::Chat.new(model: "gpt-4-turbo")
chat.with_tools(ToolOne, ToolTwo.new(foo: "bar"))
chat.ask("What is the meaning of life?")

tool_classes = RubyLLM::Test.last_request.tool_classes

assert_includes tool_classes, ToolOne
assert_includes tool_classes, ToolTwo
end
end
Loading