Implement PrepareDataPlugin for prefix cache match plugin #1942

rahulgurnani · 2025-12-04T00:52:32Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements PrepareDataPlugin hook for prefix cache match plugin. The goal is to split the prefix cache match plugin into match plugin and scorer. So, the PR adds a new scorer plugin to be used alongside the prepare data prefix cache match plugin for scoring.
This is an intermediate step towards migration of the plugin.

Did some initial scale testing using vllm:

On this version

============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  83.04     
Total input tokens:                      127000    
Total generated tokens:                  2048000   
Request throughput (req/s):              12.04     
Output token throughput (tok/s):         24663.03  
Peak output token throughput (tok/s):    52286.00  
Peak concurrent requests:                1000.00   
Total Token throughput (tok/s):          26192.43  
---------------Time to First Token----------------
Mean TTFT (ms):                          1875.19   
Median TTFT (ms):                        1877.09   
P99 TTFT (ms):                           2368.76   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          31.62     
Median TPOT (ms):                        29.59     
P99 TPOT (ms):                           39.16     
---------------Inter-token Latency----------------
Mean ITL (ms):                           31.62     
Median ITL (ms):                         27.45     
P99 ITL (ms):                            92.53     
==================================================

On head (latest release)

============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  101.31    
Total input tokens:                      127000    
Total generated tokens:                  2048000   
Request throughput (req/s):              9.87      
Output token throughput (tok/s):         20215.75  
Peak output token throughput (tok/s):    52972.00  
Peak concurrent requests:                1000.00   
Total Token throughput (tok/s):          21469.36  
---------------Time to First Token----------------
Mean TTFT (ms):                          1885.78   
Median TTFT (ms):                        1851.98   
P99 TTFT (ms):                           2482.30   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.57     
Median TPOT (ms):                        32.10     
P99 TPOT (ms):                           48.02     
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.57     
Median ITL (ms):                         27.89     
P99 ITL (ms):                            93.49

@kaushikmitr / @BenjaminBraunDev FYI

Which issue(s) this PR fixes:

Addresses #1924

Does this PR introduce a user-facing change?:

yes. this PR adds a new feature flag "prepareDataPlugins"

k8s-ci-robot · 2025-12-04T00:52:34Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2025-12-04T00:52:37Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`35eafd9`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/693784909c566800082b1d34
😎 Deploy Preview	https://deploy-preview-1942--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

liu-cong · 2025-12-04T21:26:30Z

pkg/epp/datalayer/plugins/data_types.go

+	PrefixCacheMatchPrecentKey = "PrefixCacheMatchPercentKey"
+)
+
+type PrefixCacheMatchPercent struct {


I suggest making this something more generic like PrefixCacheInfo so we can easily extend. One of things we will likely need to add is the cache tier info.

Instead (or in addition) of providing the percentage, let's provide the match length. The idea is to provide more "raw" data for consumers. Percentage is just something the scorer cares about, the latency predictor may work better with the length.

I renamed it to PrefixCacheInfo to make it easy for extension. Predictor is presently only trained to depend on prefix cache match percent. I think updating this to length may not be suitable in the short term. @BenjaminBraunDev keep me honest here. Thanks!

I think @liu-cong 's opinion carries the most weight here.

The reccomendations here seem reasonable, aggregating data for easier ergonomics of an unproven, experimental plugin, at the expense of overall usability does not seem a worth tradeoff.

I'm not sure why this comment was resolved

Updated the info to contain prefix cache related info. Thanks

rahulgurnani · 2025-12-08T15:08:35Z

/assign @kfswain

k8s-ci-robot · 2025-12-08T18:43:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rahulgurnani
Once this PR has been reviewed and has the lgtm label, please ask for approval from kfswain. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cmd/epp/runner/runner.go

pkg/epp/datalayer/factory.go

kfswain · 2025-12-08T22:16:54Z

pkg/epp/datalayer/plugins/data_types.go

@@ -0,0 +1,52 @@
+/*


consider moving this to a separate package, such as approximate_prefix

I am considering keeping all datatypes for all data layer plugins in the same place, perhaps a separate package in datalayer/plugins, something like datalayer/plugins/data_types package. Since these would be shared with the scheduling layer, this would provide better encapsulation. Let me know if that sounds good.

kfswain · 2025-12-08T22:17:46Z

pkg/epp/requestcontrol/request_control_config.go

 // If a cycle is detected, it returns an error.
-func (c *Config) PrepareDataPluginGraph() error {
+func (c *Config) PrepareDataPluginGraph(enablePrepareDataPlugins bool) error {
+	if !enablePrepareDataPlugins {


Related to the other comment, lets remove this check from here.

Done, thanks!

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer.go

kfswain · 2025-12-08T22:22:24Z

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer_test.go

+	return keys
+}
+
+func TestPrefixCacheScorer_Score(t *testing.T) {


Can we have a unit test that also creates the legacy prefix scorer, and test to make sure the created scores have identical output? We should be able to validate the behavior is the same in these unit tests

I removed the scorer from this PR since I am not using it yet. I plan to add the scorer back in a separate PR. Scoping this PR to just adding prepare data step so that predictor can use it. Thanks!

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer.go

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

cmd/epp/runner/runner.go

… comments

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Dec 4, 2025

k8s-ci-robot requested review from ahg-g and liu-cong December 4, 2025 00:52

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 4, 2025

rahulgurnani changed the title ~~Implement PreapreDataPlugin for prefix cache match plugin~~ Implement PrepareDataPlugin for prefix cache match plugin Dec 4, 2025

rahulgurnani force-pushed the pc-m branch from b8c71a8 to d6556ed Compare December 4, 2025 18:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2025

liu-cong reviewed Dec 4, 2025

View reviewed changes

rahulgurnani force-pushed the pc-m branch from c67979f to 5752e58 Compare December 6, 2025 05:15

rahulgurnani requested a review from liu-cong December 6, 2025 05:16

rahulgurnani marked this pull request as ready for review December 6, 2025 05:16

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2025

k8s-ci-robot requested review from elevran and robscott December 6, 2025 05:16

k8s-ci-robot assigned kfswain Dec 8, 2025

rahulgurnani requested a review from kfswain December 8, 2025 18:51

rahulgurnani force-pushed the pc-m branch from c330833 to ee9a486 Compare December 8, 2025 19:46

rahulgurnani mentioned this pull request Dec 8, 2025

Deprecate existing prefix cache match plugin to use the new PrepareData hook and scorer #1970

Open

kfswain reviewed Dec 8, 2025

View reviewed changes

zetxqx reviewed Dec 8, 2025

View reviewed changes

rahulgurnani force-pushed the pc-m branch from ee9a486 to 598b4b2 Compare December 9, 2025 01:23

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 9, 2025

Add PrepareRequestData method for the prefix cache plugin

396490d

rahulgurnani added 9 commits December 9, 2025 01:59

Add prefix cache match scorer

980fd29

Ensure prefix cache plugin implements all preprae data plugin methods

53c92c1

Enable prepare data plugins behind a feature flag

fdb1125

Add static check to ensure plugin implements PrepareDataPlugin interface

46b5ac2

Rename prefix cache match

1f9e0cd

Add feature gate for prepare data plugin

b3765c7

Update prefix cache match to have total and match length

a2c3da8

Add more tests, register prefix cache scorer and address other review…

cb88bdb

… comments

Move prefix cache scorer out of this PR

74cf6a5

rahulgurnani force-pushed the pc-m branch from c673022 to 74cf6a5 Compare December 9, 2025 02:03

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 9, 2025

Fix rebase errors

35eafd9

rahulgurnani requested review from kfswain and zetxqx December 9, 2025 02:49

Implement PrepareDataPlugin for prefix cache match plugin #1942

Are you sure you want to change the base?

Implement PrepareDataPlugin for prefix cache match plugin #1942

Uh oh!

Conversation

rahulgurnani commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Dec 4, 2025

Uh oh!

netlify bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani commented Dec 8, 2025

Uh oh!

k8s-ci-robot commented Dec 8, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rahulgurnani commented Dec 4, 2025 •

edited

Loading

netlify bot commented Dec 4, 2025 •

edited

Loading

rahulgurnani Dec 9, 2025 •

edited

Loading