Airflow + Helm: Simple Airflow Deployment
This is part two of a five-part series addressing Airflow at an enterprise scale. I will update these with links as they are published.
- Airflow: Planning a Deployment
- Airflow + Helm: Simple Airflow Deployment
Previously, we formulated a plan to provision Airflow in a Kubernetes cluster using Helm and then build up the supporting services and various configurations that we will need to ensure our cluster is production ready. This post will focus on getting the Helm chart deployed to our Kubernetes service. This most basic of configurations requires a database and we have chosen to use PostgreSQL in this case.
Code samples can be found here.
Preparing the Database
Here we will assume that you have:
- PostgreSQL Database Server
- Credentials with administrative access
I will be using the Azure PostgreSQL Service but any compatible version will do. First Log into the database server using the
psql "host=************.postgres.database.azure.com port=5432 dbname=postgres user=**************** password=********* sslmode=require"
Next, referring to the Airflow documentation, we can execute the following commands:
CREATE DATABASE airflow_db; CREATE USER airflow WITH PASSWORD 'your-password'; GRANT ALL PRIVILEGES ON DATABASE airflow_db TO airflow;
Pulling the Chart and Value File
After the database is set up, we can move on to preparing the chart and our values file. Using Helm, add the airflow chart repository:
helm repo add apache-airflow https://airflow.apache.org
For the values file, retrieve the default values from the chart.
curl https://raw.githubusercontent.com/apache/airflow/main/chart/values.yaml > values.yaml
Set Airflow to use the KubernetesExecutor:
Make sure we have some example DAGs to play with:
env: - name: AIRFLOW__CORE__LOAD_EXAMPLES value: "True"
Turn off the charts provided PostgreSQL resources:
postgresql: enabled: false
Input credentials and database information:
data: metadataConnection: user: airflow@some-host pass: your-password protocol: postgresql host: some-host.postgres.database.azure.com port: 5432 db: airflow_db sslmode: require
Deploying the Chart
Now that we have our values file setup for our database, we can deploy the chart. Authenticate with the cluster:
az aks get-credentials --name airflow-demo --resource-group airflow-demo
Add a namespace:
kubectl create ns airflow
The Airflow chart has a tendency towards long run times so, increase the timeout as you install the chart:
helm upgrade \ --install \ -f values.yaml \ --namespace airflow \ --timeout 30m0s \ --wait=false \ airflow \ apache-airflow/airflow
After Helm exits, we can navigate to our Kubernetes Dashboard and see the replica sets, pods, etc., that have been provisioned.
Now we should login into the cluster using the credentials provided in the Helm output. As we didn’t enable the ingress feature of the chart, access to the Airflow cluster requires port forwarding:
kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
http://localhost:8080 will bring up the login in screen. After using the credentials in the Helm output, you’ll see a table of DAGs.
To test our installation, unpause a DAG using the toggle on the left side of the screen and execute it. We expect a number of pods to be created as the tasks execute.
And that’s it, we have an Airflow cluster up and running. Now we can work on tuning the cluster to better fit our needs. The next installment in this 5-part series will handle logging in Apache Airflow!