Python gRPC Template

Updates

• 11/08/22

  • Added interactive get, put, delete implementation to the client container. Revised environment variable representation of configs. Added option flags to utils/gen_cert.sh to clarify usage of different build-time auth configurations. Updated this file with documentation.

• 10/08/022

  • First push

Usage Modes

This Python gRPC Template is designed to mitigate the boilerplate involved in setting up a gRPC server/client pair in Python, and further, in containerizing such a network. The module features some utilities to automate the most common semantics of server and client stub instantiation. The server and client code thus can be written independently of the backend connection and authentication mechanisms. These auth mechanisms include generation of build-time leaf certificates from root certificates, as well as self-signed root certificates for testing. Deployments, however, should use trusted root certificates, either obtained from a known public Certificate Authority, or generated by an internal organizational Certificate Authority.

The primary aim of this project is to allow for server and client (Docker) container nodes to be generated with minimal overhead. Commonly, gRPC APIs are overlooked in favor of REST due to the popular conception that gRPC's need for HTTP2 makes it suitable at best for internal services and inter-micro-service communication. gRPC Web is a project by gRPC which implements the usual features of language-specific gRPC packages for Javascript, and a proxy (Envoy) for web clients to access the service. This, however, is not the goal of this project. This project is intended for cases in which both endpoints are services of some kind, requiring tight integration with a Python project or environment. Client endpoints in this use case may themselves pass on functionality via normal HTTP services (i.e. Flask apps, Galaxy tools, etc.), but may also be used directly by anybody sharing the network with the gRPC servicer quite easily from, for instance, a command line, or a GUI with a backend ported to such a command line. This flexibility, modularity, and degree of native Docker + Python integration is the very goal of the project.

This template functions as an image layer which complements an already-containerized service. In this direction, the number of distinct Dockerfiles has been reduced from two to one, as the differences between the two builds were minimal, and vanished when hardcoded values were parameterized as build arguments and environment variables. Thus this project now allows building a server or client instance from the same underlying template image layer, with the two differentiated only by their build parameters. This means that client and server functionality are treated as a unit, and it is less challenging to debug such issues, as the server and client share the boilerplate configs and code that define the networking and auth functionality.

At present, the implementation has been tested on bare metal, and in Docker containers sharing a bridge network. A bridge network creates a local DNS and allows for container names to be used in referencing one another.

[NOTE: The build-time parameter COMMON_NAME should correspond to this DNS name, and is used to set the corresponding name field in any certificates that are generated. You may still use the container without providing this setting, but it is necessary if you intend to use authentication with that container].

A docker compose implementation is being developed to automate the creation and management of the network and the plugging of parameters to server and client instances. This allows for replication of nodes with potentially different image layers above and/or below the template layer, and allows for nodes across different machines (i.e. on an overlay network) to be associated in a single configuration. Additionally, such an implementation can be endowed with a set of Kubernetes Deployments and Services, and run with Docker Desktop's native Kubernetes cluster.

Security

The utils/gen_cert.sh file implements bash commands to

1. create a self-signed root certificate (to serve as a certificate authority), and
2. create new leaf certificates from a given root certificate.

Their specifications are as follows:

1. gen_root_cert [path_to_cert] [path_to_pkey] [subject], and
2. gen_signed_cert [path_to_ca_cert] [path_to_ca_key] [path_to_cert] [path_to_pkey] [path_to_cert_sign_request] [subject],

where [subject] denotes the subject field which is required in new certificates and certificate requests. When creating a root cert, minimally use /O=[owner]. When creating a leaf from this certificate, it will feature [owner] as the certificate issuer. When creating a leaf, be sure to minimally specify /CN=[common_name]. When containerized, this value must match the server's DNS name to validate the connection. Docker uses the container's name as its DNS name on user-defined bridge and overlay networks.

The bash script itself, when run with parameters, behaves a bit differently. Its specification is as follows:

sh utils/gen_cert.sh -a [AUTH] -c [CLIENT_AUTH] -h [HOST_NAME] [CA_NAME] [COMMON_NAME] [KEY]

sh utils/gen_cert.sh \
-a [ "true" | "false" ] \
-c [ "true" | "false" ] \
-h [ "[::]" | [ HOST_NAME ] \
[CA_NAME] [COMMON_NAME] [KEY]

The options (-a, -c, -h) are all required (in any order), as are their respective parameters (one each) and the three positional parameters that follow them. This script has a very specific purpose (namely, ensuring reproducible and correct implementation of certificate signing during Docker builds), so it does not attempt to handle arguments that deviate from this pattern at this time. The option parameter values following -a and -c each must each be one of "true" or "false". This is intentional, to catch any instance in which a misconfiguration has led to the script being run with faulty values. The -h parameter denotes the gRPC hostname. This value must be set to "[::]" if the certificate is for a gRPC server, or anything else if the certificate is for a client. In practice, for clients, this value is set to the COMMON_NAME of the server to which it will connect; this value is not actually used to generate the certificate (rather, its own COMMON_NAME is used). The meaning of the two flags -a and -c are as follows: , but its inequality with "[::]" indicates that a certificate should only be signed if the CLIENT_AUTH flag -c is true. which is either, should be. The remaining values are fairly self-explanatory. CA_NAME is the plaintext name of the certificate authority being used to validate connections in this container. COMMON_NAME is the plaintext DNS name used over the network for the container. KEY refers to the location of the private key with which to sign the certificate. Note that this key can be securely passed in several ways. Docker provides secret mounts during build time which are only available on the container for the duration of a single instruction. Additionally, the underlying SSL tool (openssl) supports private key encryption via passwords. This can manifest either as a request for password entry from the command line, or as a "passed-in" password via another file. This encryption may be applied both to the ephemeral authority's private key, and the internal container key which is generated alongside its signed certificate. It is recommended for production purposes that both keys be password-protected if the password can be provided securely, but it is not strictly necessary for every threat model, and may indeed present a greater risk than an unencrypted key for some setups.

References

This project was inspired in part by Dan Hipschman's Python Microservices GRPC tutorial. For security matters at a greater depth, refer to Ivan Ristić's OpenSSL Cookbook. The reader is also referred to the official OpenSSL Documentation and Python GRPC API Reference for further information, and GRPC's python examples. The GRPC examples outline many common use cases; some relevant ones include:

• helloworld
• hellostreamingworld
• auth
• route_guide