Chinese
Uncategorized

Google Cloud API Gateway Adds Streaming, but the Gateway Is Still a Boundary

Google Cloud’s API Gateway preview adds streaming for AI workloads. Forwarding a stream does not remove long-connection operations.

Jacky Wang 4 min read 152 views

When I put an AI endpoint behind a gateway, the first failure is often not the model call. A proxy closes the connection while the response is still being generated. The browser shows half an answer, while the server log records a perfectly respectable 200. Swapping models does nothing because the failure sits in the layer around the application: proxy timeouts, authentication, rate limits, and observability all need to understand that one request is still alive.

On October 21, 2026, Google Cloud announced a Public Preview for real-time streaming in API Gateway. The official post names Server-Sent Events, WebSockets, gRPC, and chunked transfer encoding. It also says the feature works without changes to existing API specifications and keeps gateway-level authentication, rate limiting, and observability in the path.

Protocol support is only the beginning

I like the direction because it moves streaming back into infrastructure design instead of leaving every service to invent a workaround. SSE is a reasonable fit for incremental text from a chat endpoint. gRPC streaming can carry multiple messages over one connection between internal services. Chunked responses may be enough when an older client already understands them.

But I would not assume that an application is ready for WebSockets merely because the gateway can forward them. Long-lived connections change the resource model. A request is no longer a function that completes in a few hundred milliseconds. It may stay open for minutes. Connection count, memory per connection, idle timeouts, reconnect behavior, and draining during a rollout all become capacity questions.

I would write down the behavior I need before tuning a timeout. For example:

streaming:
  protocol: sse
  idle_timeout: 60s
  max_duration: 10m
  retry_after_disconnect: true

That is a design sketch, not a Google Cloud configuration. The important part is separating four clocks that teams often merge into one: time to first byte, permitted idle time between events, maximum connection duration, and client backoff after a disconnect. Without those distinctions, “streaming support” is a product sentence rather than an operational contract.

The gateway still owns the stream boundary

Rate limiting also needs a different shape. One user may hold a single connection open and consume resources steadily. Another may create ten short requests. Counting requests alone hides that difference. For SSE, I would record connection start, first event, last event, clean close, and abnormal disconnect. For WebSockets, I would add ping/pong health, server close codes, and reconnect count.

Authentication cannot disappear behind a long connection either. A token may be valid when the connection starts and expired ten minutes later. A user’s permissions may change while the stream is alive. If authorization happens only once, revocation is delayed until the next connection. There is no universal answer, but the system should state whether authorization is fixed for the connection or checked again for each message.

Google’s announcement also mentions deterministic default hostnames to simplify environment provisioning and infrastructure-as-code. That sounds minor, but it is useful in practice. Addresses that depend on console state or random allocation make Terraform, preview environments, and rollback noisier. A stable hostname will not solve deployment design, but it removes one class of “the code did not change, the endpoint did” failure.

Kubernetes makes draining visible

For a streaming service on Kubernetes, readiness is not enough during a rolling deployment. A Pod can become NotReady so that new connections stop arriving while existing connections remain open. The application needs graceful shutdown: stop accepting new streams, send a clear end signal to existing clients, wait for a bounded period, and then exit. The Ingress, gateway, and application timeouts also need to agree, or the shortest layer will cut the stream first.

I would start with a small runtime test: a service emits an event every five seconds, then I observe the connection through a rollout, a short upstream failure, and a client reconnect. The useful assertions are not merely “some text appeared”. Event sequence numbers should not jump. A retry should not repeat an external side effect. The client should distinguish a clean end from an interrupted connection.

Google Cloud’s update lowers the barrier to putting streaming APIs behind a managed gateway, but it does not choose a protocol or operating model for a team. My expectation is that gateways will increasingly behave like stateful traffic junctions rather than simple HTTP route tables. For an AI product, I would start with SSE because one-way incremental output is easier to observe and operate. I would use WebSockets only when the product genuinely needs a two-way session. The longer a connection lives, the less responsibility can hide behind the phrase “the gateway supports it”.

Sources: Google Cloud streaming configuration, API Gateway release notes, Google Cloud latest announcements

发表评论