Show HN: OSS tool that let you create OpenAI DX with any LLMs
BricksLLM has native support for OpenAI, Anthropic, Azure OpenAI and vLLM. You can also integrate with other OSS LLMs using our custom provider feature.
Currently, people are using BricksLLM for the following use cases:
* Safely sharing access to different both commercial and self-hosted LLMs (e.g. teachers sharing LLM access with students, managers sharing LLM access with data scientists etc.)
* Building an AI dev platform with observability and monitoring
* Building token usage based billing
* Blocking or redacting requests that contains PII (https://github.com/bricks-cloud/BricksLLM/blob/main/cookbook/pii_detection.md)
* Improving reliability via retries, failovers and API key rotations
We believe that one underrated aspect of an LLM platform is the developer experience. The performance of a foundational model is important. However, LLMs will become less differentiated as the competition heats up among different providers. Therefore, a good developer experience is just as crucial in creating a competitive advantage. This is why I believe OpenAI has invested a lot into improving the DX of its platform. Now with BricksLLM, you can do the same to your own LLMs.
The latency of the gateway is about 15ms on my M1 Chip MacBook pro. We have load tested our applications to be able to handle more than 1000 requests/s (https://github.com/bricks-cloud/BricksLLM/blob/main/blog/how-we-built-a-highly-scalable-LLM-gateway-with-go.md).
Would love to hear your feedbacks and opinions!
Open Source Repo: https://github.com/bricks-cloud/bricksllm
Landing Page: https://trybricks.ai