🚀 Support for Multiple Endpoints in LitServe
We are introducing support for multiple endpoints in LitServe, which will require minor architectural adjustments to better separate concerns between the core components.
✨ Proposed Usage
Here’s an example of what the new API could look like:
api1 = VisionEncoderAPI(batch_size=4, api_path="/ap1")
api2 = TextEncoderAPI(batch_size=8, api_path="/path2)
def resize(...):
...
server = LitServer([
api1,
api2,
{"/random": resize}
])
server.run()🔍 Design Goals
To cleanly support this functionality, we need to ensure that responsibilities are clearly divided between the two main entities:
LitAPI
- Manages properties that pertain to the logic of individual APIs, such as:
batch_sizeloopstreamapi_path
LitServer
- Handles server-level configuration, such as:
Shared Responsibility (LitAPI + LitServer)
- Callback handling
- Streaming functionality
📌 Action Items
In PR #468, we successfully moved the batch_size parameter from LitServer to LitAPI to align with this separation of concerns. We need to apply a similar refactoring for other attributes:
🚀 Support for Multiple Endpoints in LitServe
We are introducing support for multiple endpoints in LitServe, which will require minor architectural adjustments to better separate concerns between the core components.
✨ Proposed Usage
Here’s an example of what the new API could look like:
🔍 Design Goals
To cleanly support this functionality, we need to ensure that responsibilities are clearly divided between the two main entities:
LitAPIbatch_sizeloopstreamapi_pathLitServerShared Responsibility (
LitAPI+LitServer)📌 Action Items
In PR #468, we successfully moved the
batch_sizeparameter fromLitServertoLitAPIto align with this separation of concerns. We need to apply a similar refactoring for other attributes:loopfromLitServertoLitAPI- move stream, endpoint path, loop to LitAPI initialization #512streamargument from LitServer to LitAPI - move stream, endpoint path, loop to LitAPI initialization #512api_path- move stream, endpoint path, loop to LitAPI initialization #512LitServerargument fromLitSpec.setup. Connect the queues and response buffer with a connector instead.