NOTE: I no longer use Cloudflare Tunnels for my RDS connection. They have great potential, but are the Cloudflare docs/tools/logging/etc need to mature before I keep them in production. I’ll leave this post up in case it helps someone else but would lean toward recommending a bastion or IP fixing service.
That title puts most people to sleep, but if you’re fighting through the horrendous Cloudflare docs like I was, then this post might just make your day.
Why Cloudflare Tunnels?
Most RDS tutorials have you connect directly to RDS from anywhere on the internet. This is a great way to get GuardDuty notifications every few minutes from someone trying to hack your database. So the standard fix is to limit connections to the IP address of your server. Well some of us don’t have a fixed IP. Yeah, you could use services like Fixie but there are drawbacks like cost and latency.
Next you might consider a VPN, SSH, Bastion, etc – each having its own strengths and weaknesses. But I wound up using free cloudflare tunnels for a few reasons:
- Basically free is a pretty great price
- If my database traffic has to get tunneled, I have a lot of faith in Cloudflare as the provider. Both in terms of uptime, latency, security, scalability, etc
- It’s easy to setup – (well, if you know how)
- It’s the most secure option (apart from having your server in the same private network as the db)
Getting a tunnel setup is pretty easy, but configuration becomes a nightmare because Cloudflare docs are sparse. Have I mentioned Cloudflare docs suck? Ok, no more of that.
Step 1) Setting Up Cloudflared on EC2
I’ll assume you have an AWS account, RDS is setup, and your application server is connecting to it directly.
First you need to create an EC2 instance that provides the tunnel between your AWS private network and Cloudflare. Follow the first step from this tutorial. You’ll end up with a cheap EC2 Ubuntu instance and the ability to SSH into it.
NOTE: Make sure your EC2 instance is in the same security group as the RDS instance. It needs to be able to “see” the rds instance (or whatever services your connecting to)
Second, install the cloudflared daemon on the instance & start the tunnel:
- Log in to Cloudflare Zero Trust ↗ and go to Networks > Tunnels > Create a tunnel
- Choose Cloudflared for the connector type and select Next.
- Enter a name for your tunnel (for example,
aws-tunnel) – Note: you might end up accessing multiple AWS services from this tunnel (e.g. RDS read, RDS write, etc) but this tunnel will contain all of those services. So name accordingly. - Select Save tunnel.
- Under Choose your environment, select Debian. Copy the command shown in the dashboard and run it on your AWS instance.
- Once the command has finished running, your connector will appear in Zero Trust.
- Select Next.
Now you should be able to see your “healthy” tunnel within Cloudflware > Networks > Tunnels
Step 2) Add a Hostname to Your Tunnel
You’ll need a hostname for each service that you are connecting to within your EC2 tunnel. So if your database just has one connection string right now, then you’d just need one. But if you have different connections for read/write/replica/cache/whatever then you’ll need a hostname for each of those.
NOTE: I’m 90% sure you do not need to actually create a public hostname. I believe you can connect to it later on using “UUID.cfargotunnel.com” where UUID is the Tunnel ID. But… I’m not sure how to add Access Policies against these urls.
Under Cloudflware > Networks > Tunnels, click your Tunnel’s name, then click Edit. Now click Public Hostname > “+ Add Hostname”:
- subdomain – this is the name of the service you’re connecting to
- Cloudflare will automatically create a DNS entry using the value you put here
- I tried to do sub-subdomains like “db-read.tunnel.example.com” but Cloudflare complained that it couldn’t create an SSL certificate without me paying money. Even then, not sure if it’d work.
- Again, you’ll create a host for each connection. Later on you’ll apply permissions based on these subdomains. So you may want them all to end similarly (like
db-read-tunnel.example.com)so you can apply permissions with a wildcard *-tunnel.example.com
- domain – your website’s domain
- path – fun fact, TCP connections can’t use a path so don’t waste your time. Leave it blank.
- type – tcp (yeah, you know me)
- url – if you were accessing your database from within the EC2 instance itself, what would the url be?
- Might be something like
foo-db.us-east-1.rds.amazonaws.com:5432 - If you’re not sure, ssh into that EC2 instance & test your connection string there
- Might be something like
- Advanced > Access > Protect With Access – DO NOT use this. You might think you’d use this to associate your hostname with access rules, but nope.
Step 3) Connecting From Your Server
We still need to lock some things down in both EC2 and in Cloudflare, but let’s test the connection first and make sure we’re on the right path.
There are two options for connecting to cloudflare tunnels and these get mixed up a lot.
- (1) WARP – warp is a VPN provided by cloudflare that routes all of your internet traffic through Cloudflare (this is not what we’re doing)
- (2) cloudflared tunnel – a daemon named “cloudflared” runs on your machine and connects some local port to one of the hostnames that you just setup. This is us.
So whether you’re on your development machine or server, you need cloudflared running so you can connect to your database. There’s not a special database connection string you can use to skip running this agent. You must have this agent running. To test the connection on your dev computer, try this:
# One of many ways to install it
brew install cloudflared
# Notice that you don't need to login to cloudflared
# connect your hostname to any local port
cloudflared access tcp --hostname db-read-tunnel.example.com --url localhost:9999The cloudflared access ... command blocks so you either need to run it as a background process, or open a new tab to test the connection. I’m using postgres, so here’s an easy way to test if i can list all the tables in my database:
# test the connection
PGPASSWORD="MYPASSWORD" psql "postgres://myusername@localhost:9999/mydbname" -c "\dt"Again – you’re connecting to port 9999 (which you chose above) and cloudflare magically connects that to the URL you provided within EC2.
Step 4) Tunnel Permissions
As far as the database exposure, you’re already in a better place because now only that one port is exposed. But I’m going for a layered approach so I wanted two things:
- US Only – I may not know my IP, but I do know that my servers are only in the USA. So there’s no reason for any other country to directly connect to my database.
- Token – I want a token to be required to even make this connection. Not my full blown Cloudflare login, but just something additional that prevents public connections.
To limit access to your tunnel, you need a Service Token, Application, and an Access Policy.
First create the service token via Cloudflare > Zero Trust > Access > Service Auth > Service Tokens > Create Service Token
- I like to name my tokens based on who is using it and with a year as the suffix
- Choose your duration
- Now copy down the ID and SECRET because you’ll need it later
Now create the policy. Go to Cloudflare > Zero Trust > Access > Policies > Add a Policy
- Name: I called mine “Require Token AND USA” because it requires both of these to access the tunnel
- Action: Choose Service Auth but I’ll explain the options here…
- ALLOW / BLOCK – I’m not sure if these work with TCP connections. Actually, the entire cloudflare zero trust dashboard is covered in things (like paths) that don’t work with TCP connections. Perhaps I was doing something wrong but I had Block=Everyone setup as the only rule and it still forwarded me to the Login page.
- BYPASS – If anything matches bypass then it not only skips the other policies but also skips login. (See more below)
- Service Auth – Later on you’ll learn that all Applications require a login. Like a dude sitting in a chair typing into facebook or azure or whatever. If you want people to login using a token, then Service Auth will fullfill that requirement and let you skip the login part later. Of course, none of this is documented anywhere useful but i digress..
- Rules –
- Include – Who does this rule apply to? I chose Everyone
- Require – You’ll need to add 2 requirements
- Selector: Country -> USA
- Selector: Service Token -> Choose the token you created a moment ago
- Again, this service token allows you to skip the login process
Finally, create the application. Go to Cloudflare > Zero Trust > Access > Applications > Add an Application > Self Hosted
- Name: Doesn’t matter, whatever you want
- Duration: Not sure if this matters for TCP but I chose 0
- Public Hostname:
- Input Method: Custom if you’re doing a wildcard, Default if its a single domain
- Subdomain: The subdomain from the hostname you created or “*-foo” which will cover all subdomains ending with -foo
- Domain: same as you selected earlier
- Path: leave this blank
- Access Policies – Select the access policy you just created
- Login Methods – This is the part that gets skipped because you chose “Service Auth”. If you didn’t choose a Service Auth policy, then when someone connects it will launch a browser window for them to login
Step 5) Adding Credentials To Your Connection
Now your tunnel requires a token so it’s a good time to make sure you can’t connect without a token.
NOTE: My experience is that once a tunnel is connected, changing the policy did not impact existing connections. From what I can tell, once someone is connected you can’t really kick them out.
If your cloudflared command is still running, kill it using [CONTROL]+[C]. You could also try pkill -f cloudflared
Now connect again without your credentials and run your database testing command again. My experience was that cloudflared access tcp... still starts without error, but when you actually run your database query (e.g. psql ...) then it will either fail or hang. If you’re able to query the database then something’s wrong with your policies/etc.
Now you need to add the Service Token id and secret to your connection as two headers:
cloudflared access tcp --hostname db-read-tunnel.example.com --url localhost:9999 --header CF-ACCESS-CLIENT-ID:111111111.access --header CF-ACCESS-CLIENT-SECRET:11111111111...Launch the daemon again and your database command should work now (if you’re in the US).
Now you tunnel is safe from all those canadians.
Step 6) Cloudflared In Docker
So how do you get cloudflared running on your production servers? Pretty straight forward if you control your environment. Docker Compose is another easy solution. We run on Docker containers though which are slightly tricky since Docker is supposed to be “one process per container”. There are tools like s6-overlay that let you run cloudflared as a managed service on Docker, but most recommend adding it as part of your startup/entry command. So here are the additions to my Dockerfile:
FROM node:22.11.0-alpine3.19 AS basic
# Install cloudflared
RUN wget -O /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 && chmod +x /usr/local/bin/cloudflared
# Add an entrypoint to start cloudflared
COPY entry.sh /entry.sh
RUN chmod +x /entry.sh
ENTRYPOINT ["/entry.sh"]
...
# Just for reference, this is the unchanged default command
CMD ["npm" "run" "start"]entry.sh
<strong>#!/bin/sh
</strong>
cloudflared access tcp ...... &
# run the original command
exec "$@"And of course, build and run your dockerfile and test that you can connect to the database via localhost:9999 (or whatever port you chose)
Step 7) Turn off public access to EC2 & RDS
You’ll remember we allowed any public connection to EC2 so we could ssh in and setup our cloudflare tunnel. Now you can turn off all incoming traffic because our tunnel is actually outgoing to cloudflare. Finally, the payoff!
It’s hard to say exactly what changes you should make to disable public access to EC2 & RDS, but it’s very likely that you’ll want to find the security group for your EC2 instance, make sure it’s not being used by other resources, and then disable inbound connections.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html
