Sold by: abyo software
Deployed on AWS
Serve more concurrent users and longer context per GPU. A vLLM v1 KV-cache backend that quantizes the KV cache to int4 (per-channel KEY, per-token VALUE), bit-identical to fp16 - about 3x more KV density.
Overview
S4 KV is a KV-cache backend for vLLM v1 paged attention that stores the attention KV cache in int4 instead of fp16, using KIVI-style quantization (per-channel asymmetric int4 for KEY, per-token int4 for VALUE). It is bit-identical to fp16 greedy decoding (verified on Qwen2.5 1.5B/3B/7B and Llama-3.2-3B / 3.1-8B), so you get roughly 3x more KV-cache density: more concurrent requests and longer context on the same GPU, with no measured quality loss. The image is a turnkey GPU server - an OpenAI-compatible vLLM endpoint with int4_kivi enabled - launched via the included CloudFormation template. Supported models: head_dim 128 with standard attention (Qwen2.5 and Llama-3 families); the backend fails fast at startup rather than serve incorrect output for an unsupported model. Priced on density and quality, not latency: at long context on datacenter GPUs the backend trades some decode latency for density. Runs on g5, g6, g6e, p4d, and p5 GPU instances.
Highlights
- About 3x KV-cache density vs fp16 (int4 KIVI: per-channel-asymmetric KEY + per-token VALUE) - more concurrent users and longer context per GPU.
- Bit-identical to fp16 (greedy match 1.0000, verified on Qwen2.5 and Llama-3 families) - density with no measured quality loss.
- Turnkey GPU AMI: an OpenAI-compatible vLLM server with int4_kivi enabled out of the box; launch via the included CloudFormation.
Details
Sold by
Categories
Delivery method
Delivery option
64-bit (x86) Amazon Machine Image (AMI)
Latest version
Operating system
Ubuntu 22.04
Deployed on AWS
New
Introducing multi-product solutions
You can now purchase comprehensive solutions tailored to use cases and industries.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time. Alternatively, you can pay upfront for a contract, which typically covers your anticipated usage for the contract duration. Any usage beyond contract will incur additional usage-based costs.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Cost/hour |
|---|---|
g6.xlarge Recommended | $0.113 |
g5.2xlarge | $0.17 |
g6.2xlarge | $0.137 |
p5.48xlarge | $7.706 |
p4d.24xlarge | $3.074 |
g6e.12xlarge | $1.469 |
g6e.2xlarge | $0.314 |
g6e.xlarge | $0.261 |
g5.12xlarge | $0.794 |
g5.xlarge | $0.141 |
Vendor refund policy
Software fees are billed by AWS Marketplace on your consolidated AWS bill. Refund requests are handled per the AWS Marketplace refund policy; for product-specific questions email aws-support@abyo.net .
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
64-bit (x86) Amazon Machine Image (AMI)
Amazon Machine Image (AMI)
An AMI is a virtual image that provides the information required to launch an instance. Amazon EC2 (Elastic Compute Cloud) instances are virtual servers on which you can run your applications and workloads, offering varying combinations of CPU, memory, storage, and networking resources. You can launch as many instances from as many different AMIs as you need.
Version release notes
Initial release.
Additional details
Usage instructions
Launch via the included CloudFormation template (deploy/cfn-gpu-serving.yaml): it creates the security group + IAM role and starts the vLLM OpenAI server with int4_kivi enabled. Set ModelId to a head_dim-128 model (e.g. Qwen/Qwen2.5-7B-Instruct). The OpenAI-compatible API is on port 8000 at /v1; health at /health.
Support
Vendor support
Email support: aws-support@abyo.net
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Similar products
FileMage Gateway is a FTP and SFTP server backed by Amazon S3. A web based file sharing portal is also available with customizable branding.
Liquid LFM 7B is designed to handle complex tasks, offering an optimal balance between size and output quality.
This product has charges associated with it for seller support. This VM provides you full Ethereum node with PoS (Proof-of-Stake). It has out of box setup for Ethereum which includes Geth (Go based ethereum implementation), Lighthouse (Consensus Client) and synced ledger (updated on monthly basis). The Ethereum ledger is synced till the VM publishing date making it easier and faster for you to fully sync the ledger .
Genymotion PaaS is an Android emulator on EC2. With AWS APIs, you can build or extend your own solution on top of Genymotion. Embed it into your website or product to promote your application or provide support to your sales, marketing or help desk.
Liquid LFM 40B is designed to handle complex tasks, offering an optimal balance between size and output quality.
Customer reviews
No customer reviews yet
Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.