Sep 2026 · By Somesh Motupally · 6 min read

Network Policies: Default-Deny Without Breaking Everything

The problem

Kubernetes networking is flat by default. Every pod can reach every other pod on any port — and beyond the cluster, kube-dns, the API server, and cloud metadata endpoints are all reachable unless something says otherwise. A compromised public-facing pod isn't contained to its own workload; it's a beachhead with a network view of everything.

We saw this directly during a review: a frontend pod with no business touching the data layer could open a connection to the database port, and nothing in the network path objected. The same pod could reach the cloud metadata endpoint — the classic path to credential theft in a hijacked container. None of this violated any rule, because there were no rules. The policy-as-code gates we'd built controlled what could deploy; they said nothing about who could talk to whom once it was running.

Why lockdown-by-default wasn't enough

The instinctive fix — write NetworkPolicies that allow only what we could think of, apply them everywhere — failed the first time we tried it, and failed loudly. DNS resolution broke the moment kube-dns traffic (UDP/TCP port 53) wasn't in the allow-list. Health checks started failing. Legitimate inter-service calls dropped, and the pages came in at 2 AM. Within days, teams were asking for blanket exceptions, and some policies were quietly removed to stop the bleeding — which left us worse off than before: a network policy story on paper, and a flat network in practice.

The deeper problem was that we were writing rules from guesses about traffic instead of from evidence of it. Nobody had a complete, current map of which service talks to which — and even if we had one, it would have gone stale the next sprint. There was also an uncomfortable discovery along the way: NetworkPolicies are only enforced if your CNI actually implements them. Apply a beautiful policy set on a CNI that silently ignores it, and you have the worst outcome — a false sense of containment.

The solution: default-deny, rolled out in stages

We went back at it with a staged approach: default-deny ingress and egress on every namespace, but rolled out one namespace at a time, with an observation phase before enforcement.

The rollout ran in four steps:

• Verify the CNI enforces. Before writing a single policy, we confirmed our CNI (Calico in our case) actually enforces NetworkPolicy — a quick test policy against a scratch namespace proved it. A policy that isn't enforced is a liability disguised as a control.
• Observe first, enforce second. We deployed the default-deny policies in audit/log mode first — Calico flow logs gave us a week of real traffic per namespace. That log, not our guesses, became the source of truth for the allow-lists.
• Build allow-lists from observed flows. DNS to kube-dns (port 53) was carved out everywhere on day one. Then, per workload: ingress only from the ingress controller and the specific callers the logs showed; egress only to the dependencies the logs showed — the database port, the handful of external APIs, nothing else.
• Enforce namespace by namespace. We started with the least sensitive namespaces and moved up. Each namespace got its own rollout, its own monitoring window, and a fast rollback path — so a misconfigured rule was a contained incident, never a cluster-wide outage.

None of this was one big YAML drop. The policy set is versioned alongside the rest of the platform config, and the exception process is narrow and logged: if a workload genuinely needs a new path, the request names the source, destination, port, and reason, and it expires on a timer.

Results

• Lateral movement across namespaces became structurally impossible — a compromised pod is contained to its own namespace's allow-listed paths, which are deliberately narrow.
• The metadata endpoint and database ports became unreachable from workloads that never needed them in the first place, closing off entire classes of container-escape and credential-theft paths.
• Because the allow-lists came from observed traffic, the rollout caused zero production incidents — a deliberate contrast with the first attempt.
• Exceptions stopped being silent. Every deviation from default-deny is now a logged, time-boxed, reviewed decision instead of a carve-out nobody remembers making.

Lessons learned

• DNS is the first thing that breaks. Carve out kube-dns (UDP/TCP 53) everywhere before you enforce anything else. Everyone learns this the hard way exactly once.
• Egress is where the value is. Ingress is usually already fronted by an ingress controller with its own controls. The unguarded direction is outbound — that's where data leaves and credentials get stolen.
• A week of flow logs beats a day of guessing. The observation phase was the whole difference between the failed rollout and the successful one. Traffic data is the spec.
• Roll out per namespace, not per cluster. Small blast radius per step means a bad rule is a page, not an outage — and teams build trust in the process as each namespace lands cleanly.
• A gate without an exception path gets bypassed. Same lesson as the admission gates: a narrow, logged, time-boxed override process keeps people going through policy instead of around it.
• Verify enforcement before you trust it. A NetworkPolicy on a CNI that ignores it is decoration. Test it in a scratch namespace and watch the drop happen.

What's next

IP-and-port rules get us containment, but they're still coarse: they say nothing about which workload is on the other end of an allowed connection. The next step is identity-aware policy — using workload identity (via the service mesh's mTLS identities, or Cilium's identity-based policies) so rules say "the checkout service may talk to the payments API" instead of "anything in namespace A may reach port 443 in namespace B." That's the move from network segmentation to real zero-trust inside the cluster — and the SLSA provenance work from the previous post's roadmap is still queued behind it.

If you've rolled out default-deny NetworkPolicies without taking down production, I'd like to hear how you handled the observation phase — get in touch.

Part of the Pipeline to Runtime series.