What is Multi-Tenancy

just a note

What is tenant?

• A person who occupies land or property rented from a landlord.
• A consumer who consumers services (as SaaS, IaaS or *aaS) offered by Cloud service Providers.

Multi-Tenancy

• Have multiple tenants/occupants.
• Software that is shared by several consumers.
• Each consumers is unaware of the others.

Approaches

• Infra Level Tenancy
• Schema Level Tenancy/Shared Services (API)
  • DB sharding
  • Approaches Tenant, Service, Geo, etc
  • Tenant_id, Tenant_name, User_login, Location

Kubernetes Multi-tenancy

Providing isolation and fair resource sharing between multiple users and their workloads within a cluster.

It’s important that it is in a single cluster.

Trust

• Trust, Semi-Trust, Non-trusted

Kubernetes Engine Multi-Tenancy Primitives

• Cluster vs Namespace boundary

Cluster per Tenant

• Pros
  • Separate control plane (API) for each tenant (for free)
  • Strong network isolation (if it;s per-cluster VPC)

Namespace per tenant(intra-cluster multi-tenancy)

Namespaces provide logical isolation between tenants on a cluster.

Kubernetes policies are namespace-scoped.

• Logical isolation between tenants
• Policies for API access restrictions & resource usage constraints
• Pros
  • Tenants can reuse extensions/controllers/CRDs
  • Shared control plane(=shared ops, shared security/auditing)

Kubernetes engine primitives

• Access Control
  • Admission Control
  • RBAC (Role-based access control)
  • IAM (Identity and Access Management)
• Resource Sharing
  • Quotas
  • Pod Priority
  • Pod Affinity/Anti-Affinity
  • Limit Range
• Runtime Isolation
  • Sandbox Pods
  • Pod Security Policy
  • Network Policy

Use cases

• Enterprise
  • All user s from the same company/organization
  • Namespaces=Tenants=Teams
  • Semi-trusted tenants
  • Cluster Roles
    • Cluster Admin
      • CRUD any policy objects
      • Create/assign namespaces to Namespace Admin
      • Manage policies (resource usage quotas, networking)
    • Namespace Admin
      • Mange users in the namespace they own
    • User
      • CRUD non-policy objects in the namespace(s) they have access to
  • Many apps from different teams, semi-trusted
    • Vanilla container isolation may suffice
  • Network isolation
    • allow all traffic within a namespace
    • Whitelist traffic from/to other namespaces(=teams)
• Software as a Service
  • Consumer deploys their app through a custom control plane
  • SaaS API is a trusted client of Kubernetes
  • Cluster admins can access the Kubernetes API directly.
  • Tenant workloads may have untrusted pieces:
    • such as WordPress extensions
    • may require sandboxing with gVisor etc
• Kubernetes as a Service
  • Untrusted tenants running untrusted code.
    • Platform as a Service or hosting companies
  • tenants may create their namespaces, but canot set policy objects.
  • Stronger isolation requirements than enterprise/SaaS.

Kubernetes engine multi-tenancy primitives

• Auth related
  • RBAC (Role-based access control)
    • which users/groups/Service Accounts
    • can do which operations
    • on which API resources
    • in which namespaces
    • mostly useful for
      • giving access to pods calling Kubernetes API (with Kubernetes Service Accounts)
      • giving fine-grained access to people/groups calling Kubernetes API (with Google accounts)
  • IAM (Identity and Access Management)
    • practical for giving Google users/groups project-wide access
  • Admission Control
    • Intercept API request before resource is persisted
    • can mutate and allow/deny
    • compiled into Kubernetes apiserver binary
    • Extending Admission Controls
      • can develop webhooks to create your own Admission Controllers
  • Pod Security Policy
    • Restricts access to host{filesystem, network, ports, PID namespace, IFS namespace}…
    • Limits privileged containers, volume types, enforces read-only filesystem etc
    • Enforced through its own admission plugin
  • Network Policy
    • Which pods can talk to which other pods(based on their namespace/labels) or IP ranges.
    • Available on Kubernetes Engine with Calico Network Plugin
• Scheduling related
  • Pod Priority
    • Puts high priority pods waiting in pending state in front of the scheduling queue
  • Pod Preemption
    • Evicts lower priority pods from a Node, if high priority pod cannot be scheduled due to not enough space/resources in the cluster
  • Resource Quotas
    • Limits total memory/cpu/storage that pods can use, and how many objects of each type(pods, load balancers, ConfigMaps, etc.) on a per-namespace basis
  • Pod Affinity/Anti-Affinity
    • constrain scheduling of pods, baszed on the labels of other pods scheduled on the node
  • Limit Range
  • Sandbox Pods
    • Linux kernel bugs and security vulnerabilities may bypass container security boundaries
      • Kata Containers
      • gVisor(google approach)
  • Pod Security Context

Kubernetes Multi-tenancy Limitations Today

• in 2018
• Kubernetes API
  • Currently API calls are not rate limited, open to DoS from tenants, impacting others.
• Networking:
  • Networking is not a scheduled resource in Kubernetes, yet(cannot use with limits/requests)
  • Tenants can still discover each other via Kubernetes DNS

Key Takeaways

• Determine your use case
  • How trusted are your tenant users and workloads?
  • What degree and kinds of isolation do you need?
• Namespace-centric multi-tenancy
  • Utilize Policy objects for scheduling and access control
  • Think about personas and map them to RBAC cluster roles
  • Automate policies across clusters with GKE Policy Management(alpha)