Apple Push Notifications with Kubernetes and Pub/Sub

This is how I implemented server side Apple Push Notifications by writing a service in Go and deploying it on Container Engine (Kubernetes) with Cloud Pub/Sub as the worker queue.
The architecture can be used to scale production work loads for sending notifications but also as a guide on how to distribute similar work loads in general.

Apple Push Notifications

Apple provides a modern and simple API for sending notifications to registered device tokens. How to generate these tokens (among other things) can be found in the Remote Notification Programming Guide.
To authenticate with Apple, you need to chose between a token based or a certificate based connection trust. Both methods also require your client to support HTTP/2. Each notification is then delivered as a JSON payload to this endpoint.
In this article, we chose the certificate based trust. Luckily for us, HTTP/2 is already a supported protocol by the standard Go library.

Google Cloud Pub/Sub

Cloud Pub/Sub is a fully managed service for scalable real-time messaging. It allows you to decouple different systems by putting a message queue in between. The Pub/Sub service is guaranteed to deliver published messages at least once to one or more subscribers.
We can then have our service subscribe to messages for a topic that push notification messages are published to. Messages doesn’t necessarily have to be published from servers residing on Google networks. As long as we provide credentials to the publisher, it can be from anywhere on the internet.

Deploying with Google Container Engine a.k.a. Kubernetes

Go is a great language for creating small (and large) services and allows us to create highly efficient subscribers to consume messages to be sent from the Pub/Sub topic. This is also the language that Kubernetes and Docker is developed in.
See the full source at github for this subscriber deployment. I’ve setup a build process for this repository through the docker hub, you don’t have to build the container yourself. It’s publicly available as joonix/apn.
If you don’t already have a kubernetes cluster, you may create one using the gcloud command.

gcloud cluster create yourclustername --scope=cloud-platform

It is important that the cluster is allowed access to the correct scopes when creating it or it won’t be able to access the Pub/Sub service. You may also configure the access keys manually, but that is outside of the scope of this article (pun intended).
The next step is to provide our Apple developer certificate. This is required to authenticate with the push notification API when sending notifications. We do this by storing the contents in what is called secret objects in the kubernetes environment. This way, we won’t risk checking in the sensitive information to our git repository or other means of file sharing. How to retrieve these certificates in the first place is part of the documentation provided by Apple.

kubectl create secret tls apple-developer --cert cert.pem --key key.pem

The remaining configuration is less sensitive and can be set by using a config map. We need to set the Google Cloud project name for Pub/Sub as well as the bundle ID associated with our developer certificate.

kubectl create configmap apn --from-literal=project=<your-gcp-project> --from-literal=bundle=<your-apple-bundle-id>

Finally we can deploy our service, I’ve created an example deployment that can be used with kubectl. It will read the above configurations and certificates and provide it as parameters and files for our apn service. There shouldn’t be any additional modifications required, you simply let kubectl load the deployment into the cluster:

kubectl create -f deployment.yaml

Testing it out

To test that your service is able to consume messages from the Pub/Sub topic and send them as a notification through Apple, you can use the integration test example.
As the test needs access to the real thing, you will need to authenticate.

gcloud auth application-default login

This makes the API credentials available similar to when running on container engine, no additional credentials needs to be provided to the application.
Run the test with provided device token and project information:

go test -tags integration -token="<token retrieved from your app>" -project="<your-gcp-project>" -topic="notifications" -run TestPubsubNotificationIntegration

A message will be published on the specified topic which can hopefully be consumed by the apn service. If everything goes well, you should see the notification on the target device.

Last words

For docker hub compatibility reasons, I’ve made the container depend on the golang docker image. As this image includes a full Go installation and dependencies for compiling, it becomes much larger than necessary.
In a production environment where you might care about image size and start up times, you could instead use a 2-step build process. First step for compiling the binary and second step for building a much smaller container that only contains this binary. The Dockerfile in the second step could use FROM scratch or the popular FROM alpine. Alpine enables other useful packages that might be necessary such as root certificates for TLS, without adding too much overhead as other popular distributions would.