Start Now Login
How We Use AWS IAM to Generate Temporary Amazon Redshift Passwords

How We Use AWS IAM to Generate Temporary Amazon Redshift Passwords

At intermix.io, we run a fleet of over ten Amazon Redshift clusters. In this post, I’ll describe how we use IAM authentication for our database users. By using IAM credentials, we can enforce security policies on users and passwords. It’s a scalable and secure way to manage access to your cluster(s).

The approach we describe is useful in environments where security is a top concern. Examples are industries with regulatory requirements such as Finance or Healthcare. Other use cases include enterprises with strict IT security compliance requirements.

————

Secure ways to manage access to Amazon Redshift

Amazon Redshift started as a data warehouse in the cloud. A frequent initial use case was business intelligence. Over the years though, the use cases for Amazon Redshift have evolved. Redshift is still used for reporting. But it’s now also part of mission-critical data services and applications. intermix.io is an example – we built our own application on top of Redshift. And so it’s critical to track and manage user access to a cluster.

The common way to for users to log onto Amazon Redshift is by providing a database username and password. This is due to the fact that Amazon Redshift is based on PostgreSQL.

But using a username / password combination has a few drawbacks.  The biggest one is that there is no way to enforce security best practices for password changes.

This may not be an issue when you are in a small company where some 2-3 people have access to your cluster. But that’s different for enterprises with a large pool of users. It’s also different when you’re a small company where headcount is growing fast. It’s easy to lose track of everybody who has access to your most valuable data sets.

Luckily, AWS has recently developed alternatives to using a username and password. There are three options to maintaining login credentials for your Amazon Redshift database:

  1. Permit users to create user credentials and login with their IAM credentials.
  2. Permit users to login with a federated sign-on sign-on (SS) through a SAML 2.0-compliant identity provider.
  3. Generate temporary database credentials. Permissions are granted through an AWS Identity and Access Management (IAM) permissions policy. By default, these credentials expire after 15 minutes but you can configure them to expire up to an hour after creation.

This post will discuss #3: using IAM credentials to generate expiring database credentials.

For programmatic generation of temporary or auto-expiring user credentials, Amazon Redshift provides the get-cluster-credentials command for the AWS Command Line Interface (AWS CLI). The CLI uses the API endpoint GetClusterCredentials, which you can also call directly. GetClusterCredentials returns a database user name, a temporary password and a temporary authorization.

How intermix.io uses IAM to Generate Temporary Passwords

One of my first tasks on joining intermix was to update the way we enforce credential rotation for our clusters.

The old way – a manual process

Handling credentials used to be a manual process, and that’s a pain in the neck. We’re already using the AWS Secrets Manager. It could have been a pretty trivial exercise to add auto-rotation to our secrets and then trigger ALTER <user> queries to update them with new credentials. That would have been my initial approach. But one of my new colleagues pointed out the option of using IAM. Redshift allows you to get time-scoped IAM credentials associated with a role within Redshift itself.

The new way – using IAM to generate expiring passwords

That turned into a somewhat larger undertaking. First I had to understand how we were using Redshift across our platform. With a reasonable idea of how the change would work, I had to change our Redshift connection logic to pull credentials before connecting. That turned out to be a pretty easy change. We’re a Python shop and Boto3 – the AWS SDK for Python – is exhaustive. Boto enables Python developers to create, configure, and manage AWS services.

We deployed the change into one of our testing environments. Everything went well until we hit the rate limit for the Redshift API.  We were making too many requests to GetClusterCredentials. We have a lot of in-flight transformations that all connect to Redshift. Making those calls along-side the connect exhausted the rate-limit. But that wasn’t insurmountable. It was pretty easy to add a caching mechanism to our connection logic so that we didn’t need to generate a new credential every time.

Once we got the caching mechanism deployed, we were able to disable logins. Access to the cluster was now only available through IAM credentials.

That left us with an awkward situation, though. Our own developers have to connect to our clusters to run queries or test new features. They needed a way to generate IAM credentials and connect to a remote cluster.

We already understood how to generate IAM credentials and had code that handled that. We then solved our need by creating a task in our execution and deployment framework. The task connects to a Redshift cluster in any of our environments using IAM credentials. You can see it in use below!

The code we used to do this isn’t actually that important because it’s made possible by the Redshift API and some IAM policies. So you can do this yourself, even without seeing what I’ve done. You need an AWS client for your preferred programming language, and an IAM policy profile granting your users access to get IAM credentials on your clusters.

Getting started

The first thing you’re going to want to do is to create an IAM policy with the Redshift Actions needed to create an IAM credential on a Redshift cluster. You can set this to be as restricted as you’d like, but for the purposes of this blog post we’re not going to restrict it at all.

Once you have that policy created, go ahead and create a group to contain the users to whom you want to grant access. If you’ve already got that group created, great! Attach the policy you defined in the previous step. Add the users you’d like into the group and attach the policy under the Permissions tab.

At this point, all users will now be able to generate IAM credentials for existing users on your clusters. But using the AWS CLI is manual and can be error-prone. My recommendation is to create a utility to generate the credentials and connect to the cluster on behalf of the user. This utility could also generate ODBC or JDBC connection strings if that’s how your users connect to a cluster.

Here’s a quick Python-based example that outputs a JDBC URL. The examples assume that:

You’re now in a situation where you have an IAM policy and a group containing your users. You’ve also configured your Redshift users with passwords DISABLED.

In short, you’ve secured your Redshift cluster. Your security team can enforce credential rotation on users using standard IAM behavior vs. enforcing them on the database itself.

I hope that helps!

For more on best practices when working with Amazon Redshift, read our post on 3 Things to Avoid When Setting Up an Amazon Redshift Cluster. Or download our best-practices guide for more tips on enterprise-grade deployments for Amazon Redshift.

Ready to start seeing into your data infrastructure?
Get started with a 14-day free trial, with access to the full platform

No Credit Card Required