Japan Just Named the Kubernetes AI Stack: What a Self-Hosted PaaS Should Adopt, Pilot, or Skip

Japan just gave the Kubernetes AI stack a name, a scope, and a meeting cadence. On July 23, 2026, a CNCF blog post by Shingo Omura (Principal Architect of AI Infrastructure at LY Corporation) and Sunyanan Choochotkaew (CNCF Ambassador, IBM Research Tokyo) launched the AI Infrastructure SIG under the CNCF Japan chapter — with organizers from LY Corporation, IBM Research Tokyo, CyberAgent, Preferred Networks, and Fsas Technologies, a six-layer technical scope, and a first meetup on October 1, 2026 in Tokyo. For a self-hosted PaaS team staring at a menu of DRA, Kueue, KubeRay, KServe, llm-d, Gateway API Inference Extension, agent sandboxes, and AI conformance, the verdict up front is this: adopt three now, pilot two, skip two stacks without guilt, and watch two with dated triggers. No project's maturity moved because it got named — but the triage order just got a lot clearer.

The rest of this post earns every row.

What the SIG actually is​

The launch post is admirably concrete, so let's stay concrete. The AI Infra SIG lives under Cloud Native Community Japan, not under the global CNCF TOC — it is a regional practitioner group, and its stated mission is sharing operational best practices, optimization techniques, and upstream contribution experience for AI workloads. Its six organizers, listed alphabetically in the announcement, are Masaya Aoyama (CyberAgent), Sunyanan Choochotkaew (IBM Research Tokyo), Toru Komatsu (Preferred Networks), Shingo Omura (LY Corporation), Kenta Tada (CNCF End User TAB and eBPF Foundation Governing Board), and Kenji Tagashira (Fsas Technologies). That roster matters: these are operators of some of Japan's largest production fleets and researchers shipping upstream, not a vendor alliance.

The technical scope comes in six named layers: scheduling (DRA, workload-aware scheduling, Kueue), orchestration (JobSet, LeaderWorkerSet, KubeRay), AI deployment platforms (KServe, llm-d, NVIDIA Dynamo, AIBrix), networking (Gateway API Inference Extension, kgateway, Envoy AI Gateway), agent infrastructure (agentgateway from the Agentic AI Foundation, Agent Sandbox), and standards and conformance (the AI conformance effort under SIG Architecture). Planned joint work includes CoHDI Japan on the DRA front and collaboration with the PyTorch Foundation and the Agentic AI Foundation — with AGNTCon plus MCPCon Japan landing in September 2026 as the adjacent event to watch. The first meetup, themed "Let's Get Started with the CNCJ AI Infra SIG — Where Are We Today?", runs October 1, 2026, 18:00–21:00 JST in Tokyo with livestreaming, and speaker proposals were open through August 28.

Two things to notice. First, the scope deliberately spans training and inference, batch and serving, infrastructure and agents — the whole surface, named as one roadmap for the first time by a vendor-neutral group. Second, naming is not maturing: every project on the list is exactly as production-ready as it was on July 22. The SIG changes the coordination around the projects, not the projects. Keep that separation and the table above almost writes itself.

Adopt now: DRA core, the Inference Extension API, and Kueue​

Start with Dynamic Resource Allocation, because it is the furthest along and the most load-bearing. DRA core — ResourceClaims, DeviceClasses, the resource.k8s.io/v1 API — reached general availability in the 1.34/1.35 window, and Kubernetes 1.36 ("Haru," released April 22, 2026) hardened the story further: prioritized list ordering went stable and the ecosystem followed, with NVIDIA moving its DRA GPU driver into Kubernetes SIG governance and managed providers like DigitalOcean shipping DRA drivers for GPU node pools in preview on 1.36. For a self-hosted fleet, the concrete win is retiring the nvidia.com/gpu integer-count hack: workloads request accelerators through structured claims the scheduler actually reasons about, which is the precondition for everything else in this post — bin-packing inference onto shared GPU nodes, topology-aware placement, per-tenant device isolation. If you run Cluster API on owned machines, DRA core belongs in your next minor-version upgrade plan, full stop.

Next, the Gateway API Inference Extension — adopt the API now, defer the implementation choice. Version 1.0 delivers stable InferencePool and related routing CRDs plus an endpoint picker, typically wired through Envoy's external-processing filter, that routes inference requests with model awareness: prefix-cache affinity, latency SLO objectives, per-model backends. Google's own writeup makes the cost case crisply — naive round-robin across model servers triggers expensive accelerator recomputation and latency spikes, while cache-aware routing reuses KV-cache prefix state. Both Envoy AI Gateway and GKE's Inference Gateway have pinned to v1.0, and GKE ships an inferencepool v1.0.0 chart, so the API surface you program against is stable even as implementations compete.

For a PaaS, this is the difference between "we proxy to a model" and "we route to models like a platform": adopt the CRDs and the routing model now, and pick kgateway versus Envoy AI Gateway when your first real inference pool goes live, using v1.0 conformance and ext-proc maturity as the selection test.

Third, Kueue — the quietest adopt on the list and arguably the most immediately useful. Kubernetes' default scheduler has no notion of queues: whoever's pods arrive first grabs capacity, and a bulk agent-eval run can starve interactive tenant workloads (or vice versa) with no fairness policy anywhere. Kueue adds k8s-native job queueing with ClusterQueues, quotas, preemption, and increasingly multi-cluster scheduling, which maps exactly onto the workload a PaaS fleet actually has: batch-style agent sandbox runs, eval suites, and bulk provisioning jobs competing with latency-sensitive web services on the same nodes. Start with a single ClusterQueue and a quota split between "interactive tenant work" and "batch agent work" — that one policy buys more operational calm than any dashboard.

Pilot: DRA's beta features and workload-aware scheduling​

Kubernetes 1.36 graduated DRA's core but shipped its most interesting capabilities as beta, enabled by default yet still stabilizing: partitionable devices (slicing one GPU into claimable fractions), consumable capacity (tracking non-integer device resources), and device binding conditions (readiness gates before scheduling). This is the machinery that turns "one GPU per tenant" into "tenants share a GPU pool with fractional claims" — the utilization story that makes owned GPU nodes pencil out for a small fleet. Pilot it the honest way: stand up one heterogeneous or GPU-bearing node pool, exercise fractional claims and binding conditions against a staging inference workload, and track upstream breakage per 1.36-to-1.37 release note before any tenant touches it.

Workload-aware scheduling rides alongside: the kube-scheduler learning to treat a multi-pod workload as one placement unit rather than N independent pods. On a homogeneous Hetzner-style CPU pool running stateless web services, expect near-zero visible win — say so in the pilot doc, so nobody declares victory over noise. The pilot earns its keep the moment your pool stops being uniform: mixed CPU/GPU nodes, topology-sensitive inference, or prefill/decode splits across hardware. Until then it is a lab, not a default.

Skip without guilt: training orchestration and the full serving stack​

Here is where the SIG's one-scope framing is actively misleading for a PaaS team, and where scope discipline pays. JobSet, LeaderWorkerSet, and KubeRay orchestrate distributed training: gang-scheduled, all-or-nothing, multi-node runs that scale up and down badly by design. Your fleet runs git-push web apps and short-lived agent sandboxes — workloads with the opposite shape. Adopting training orchestration "because the SIG lists it" is scope creep with a YAML installer. Note the contrast the same season provided: Kubeflow graduated in CNCF in August 2026 after incubating since 2023, explicitly as the standard for ML-pipeline operations — a mature, available, and entirely skippable stack for a team that doesn't run ML pipelines. The flip condition is a tenant request for multi-node training, and not before.

Same verdict, sharper edge, for the serving stack: KServe, llm-d, NVIDIA Dynamo, AIBrix. Each is a real answer to "how do I serve models at scale" — autoscaling inference backends, disaggregated prefill/decode, KV-cache-aware load balancing. But running a model-serving platform is a second product: model lifecycle, weight storage, per-model SLOs, accelerator capacity planning. A PaaS whose tenants bring their own model endpoints needs inference routing (the Extension, adopted above), not inference serving. The day you host tenant weights yourself, re-open this row and pick among the four with a bake-off; until then, every serving-stack install is operational surface area with no tenant behind it.

Watch with triggers: agent infra, conformance — and what the name actually changed​

The last two rows are watches, and a watch without a trigger is just procrastination — so here are the triggers. Agent infrastructure (agentgateway under the Agentic AI Foundation, the Agent Sandbox project) is standardizing the fastest-moving target in the industry: how agents authenticate, delegate, and execute in sandboxes. Adopting pre-stability agent APIs risks a rewrite per quarter; instead watch two dated signals — technical content from the SIG's October 1 meetup and AGNTCon plus MCPCon Japan in September 2026 — and adopt when the sandbox API holds still across two consecutive releases. AI conformance is even simpler: there are no published "AI Ready" criteria to conform to yet, so track SIG Architecture's conformance effort and schedule your evaluation for the week criteria land, not before.

Which answers the TODO's second question directly: the SIG changed the coordination, not the maturity. No version number moved because six organizers named six layers. What moved is real anyway — a vendor-neutral room where an LY Corporation fleet operator, a Preferred Networks engineer, and an IBM researcher triage the same stack your roadmap depends on, on a meetup cadence, in the open, with joint sessions against CoHDI, PyTorch, and the agent foundations. For a small self-hosted team that cannot attend every upstream community, that room is a filter: read the October 1 talks the way you'd read release notes, and let a regional SIG do the horizon-scanning your headcount can't.

The stack finally has a name. Your roadmap now has an order: DRA core, the Inference Extension, and Kueue this half; DRA beta in the lab; training and serving stacks closed until a tenant reopens them; agents and conformance on a dated watch. That is what a coordinated scope is for — not adopting everything, but knowing exactly what you're declining.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.