Routing may account for health, queue length, model version, session affinity, or hardware capacity. It removes failed instances and helps prevent a small subset from becoming overloaded.
A load balancer distributes incoming inference requests across available service instances.
Routing may account for health, queue length, model version, session affinity, or hardware capacity. It removes failed instances and helps prevent a small subset from becoming overloaded.