post originally made to https://ii.nz/post/rerouting-container-registries-with-envoy

Introduction

In this post, I will detail the discovery of Envoy’s dynamic rewriting location capabilities and the relationship to OCI registries.

What is Envoy?

open source edge and service proxy, designed for cloud-native applications

What is an OCI container registry?

a standard, and specification, for the implementation of container registries

I’ve been playing around with and learning Envoy for a number of months now. One of the concepts I’m investigating is rewriting the request’s host. Envoy is a super powerful piece of software. It is flexible and highly dynamic.

Journey

My expectations

The goal is to set up Envoy on a host to rewrite all requests dynamically back to a container registry hosted by a cloud-provider, such as GCP.

Initial discoveries

One of the first things I investigated was the ability to get traffic from one site and serve it on another (proxying). I searched in the docs and in their most basic example could see that, by using envoy’s http filter in the filter_chains, a static host could be rewritten.

Example:

...
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /dev/stdout
          http_filters:
          - name: envoy.filters.http.router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite_literal: www.envoyproxy.io
                  cluster: service_envoyproxy_io
...

This is a great start! This serves the site and its content under the host where Envoy is served. However, the host in the rewrite is static and not dynamic. It seems at this point like doing the implementation this way is not viable.

Learning about filter-chains

Envoy has the lovely feature to set many kinds of middleware in the middle of a request. This middleware can be used to add/change/remove things from the request. Envoy is particularly good at HTTP related filtering. It also supports such features as dynamic forward proxy, JWT auth, health checks, and rate limiting.

The functionality is infinitely useful as filters can be such things as gRPC, PostgreSQL, Wasm, and even Lua.

The implementation

Once I found the ability to write Lua as a filter, I found that it provided enough capability to perform the dynamic host rewrite.

static_resources:
  listeners:
  - name: main
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          codec_type: auto
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: web_service
          http_filters:
          - name: envoy.filters.http.lua
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
              inline_code: |
                local reg1 = "k8s.gcr.io"
                local reg2 = "registry-1.docker.io"
                local reg2WithIP = "192.168.0.1"
                function envoy_on_request(request_handle)
                  local reg = reg1
                  remoteAddr = request_handle:headers():get("x-real-ip")
                  if remoteAddr == reg2WithIP then
                    request_handle:logInfo("remoteAddr: "..reg2WithIP)
                    reg = reg2
                  end
                  if request_handle:headers():get(":method") == "GET" then
                    request_handle:respond(
                      {
                        [":status"] = "302",
                        ["location"] = "https://"..reg..request_handle:headers():get(":path"),
                        ["Content-Type"] = "text/html; charset=utf-8",
                        [":authority"] = "web_service"
                      },
                      '<a href="'.."https://"..reg..request_handle:headers():get(":path")..'">'.."302".."</a>.\n")
                  end
                end                
          - name: envoy.filters.http.router
            typed_config: {}

  clusters:
  - name: web_service
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    lb_policy: round_robin
    load_assignment:
      cluster_name: web_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: ii.nz
                port_value: 443

With envoy running this config, the behaviour of the requests is

  • rewrite all traffic hitting the web service to k8s.gcr.io
  • except if the IP is 192.168.0.1 then set the location to registry-1.docker.io

Since I’m using a Pair instance, it sets the local subnet to 192.168.0.0/24 so when I try to docker pull humacs-envoy-10000.$SHARINGIO_PAIR_BASE_DNS_NAME/library/postgres:12-alpine it will go to docker.io.

On my local machine, pulling container images using docker pull humacs-envoy-10000.$SHARINGIO_PAIR_BASE_DNS_NAME/e2e-test-images/agnhost:2.26 will instead use k8s.gcr.io.

To achieve this, I research how other http libraries handle redirects - namely Golang’s net/http.Redirect. The main things that Golang’s http.Redirect does is:

  • set the content-type header to text/html
  • set the location to the destination
  • set the status code to 302
  • set the body to the same data in earlier steps, but in an a tag.

Final thoughts

I’m learning that Envoy is highly flexible and seemly limitless in it’s capabilities.

It’s exciting to see Envoy being adopted in so many places - moreover to see the diverse usecases and implementations.

Big shout out to Zach for pairing on this with a few different aspects and attempts! (Zach is cool™️)