Elastic Node Resizing in Redshift
Amazon Redshift is a powerful, feature-rich data warehouse solution that has been acclaimed for its scalability. But while scaling up is a crucial functionality in order to support growing business needs, scaling down is just as important. This is where elastic resizing across node types can help.
Redshift users don’t always require the same compute and storage capacities every day, or at every hour of the day. For example, cluster capacities can often be scaled down at nighttime or on the weekend. What’s more, Redshift compute and storage requirements may need to be rapidly scaled up or down based on unpredictable behavior (e.g. the arrival of new data from a network of weather sensors).
The good news is that users now have more options than ever for quickly and efficiently adjusting their usage of the Redshift platform. In April, Redshift announced that it would support elastic node resizing. So what does this mean for Redshift users, and how does this new feature perform in practice?
Table of Contents
- What is Elastic Resizing in Redshift?
- How Does Elastic Resizing Work?
- Redshift Elastic Resizing Across Node Types
- Elastic Node Resizing in Practice
What is Elastic Resizing in Redshift?
The Redshift elastic resizing feature was first announced in November 2018. According to the Redshift website, the feature allows users to “resize your Amazon Redshift cluster in minutes by adding nodes to get better performance and more storage for demanding workloads, or by removing nodes to save cost.”
Elastic resizing represents a new alternative for users to resize their Redshift clusters. The two other Redshift resizing options are:
- Classic resizing: Classic resizing allows users to change the number of nodes in the cluster and/or the node type. With classic resizing, the tables in the original cluster are replicated in a new cluster. The original cluster will remain in read-only mode until the operation is complete.
- Snapshot, restore, and resize: During elastic resizing and classic resizing, the original cluster is unavailable or read-only, respectively. If you want to maintain access to the cluster during the resize operation, you can instead take a snapshot of the original cluster and resize the new snapshot. This method maintains access to the original cluster, but in order to guarantee consistency, you will need to replicate any write operations made to the original cluster before the resizing is complete.
Below, we’ll investigate the elastic resizing feature in more detail, as well as the new ability to perform Redshift elastic resizing across node types.
How Does Elastic Resizing in Redshift Work?
Elastic resizing in Redshift allows users to quickly add or remove nodes from a Redshift cluster. Rather than creating a new cluster, as is done with classic resizing, elastic resizing modifies the number of nodes in the original cluster. The steps of elastic resizing in Redshift are as follows:
- Step 1: Redshift takes a snapshot of the original cluster. This includes no-backup tables such as staging tables that are not usually included in cluster snapshots. The snapshot operation will be faster if you have enabled automatic snapshots, or if you have recently taken a manual snapshot.
- Step 2: Redshift begins the resizing operation. This step usually lasts no more than a few minutes. The existing workload on the original cluster is put on hold: Redshift holds open the current session connections, and queries are placed in a queue. During this stage, the cluster does not accept new connections or queries, and some existing sessions and queries may time out.
- Step 3: Redshift restores session connections and allows queries to proceed as normal. Meanwhile, Redshift redistributes data to the new node slices in the background. The cluster may be slower than usual until the replication process is complete.
Amazon claims that in most cases, elastic resizing will complete in a manner of minutes, which makes it significantly faster than a classic resizing operation. However, the speed of elastic resizing in practice will depend on a number of factors, such as:
- The existing workload on the original cluster.
- The number and size of the database tables that will be replicated.
- The distribution of data across the different compute nodes in the cluster.
To speed up an elastic resizing operation, Amazon recommends a number of Redshift best practices, including:
- Using Redshift’s Table Skew Inspector SQL script in order to identify database tables that are “skewed,” i.e. distributed unevenly between different node slices. Fixing skewed tables can help improve the parallelism of the resizing operation.
- Deleting data and tables that are unused and unnecessary.
Some of these recommendations are specific to the elastic resizing operation, while others are general best practices for Redshift. To learn more about optimizing your Redshift deployment, check out our article “Top 14 Performance Tuning Techniques for Amazon Redshift.”
Redshift Elastic Resizing Across Node Types
The original elastic resizing operation was introduced in November 2018. Although it represented a significant advancement over classic resizing, elastic resizing was still limited by the fact that users could not modify the node types within the cluster.
All that has changed as of April 2020, when Amazon announced the introduction of Redshift elastic resizing across node types. This new feature is available for Redshift release versions release version 1.0.10013 and later.
Users can now alter not only the number of nodes, but also the node types with a simple operation that completes in a matter of minutes. For example, users can upgrade to Redshift’s new RA3 node type introduced in December, which lets you scale a cluster’s compute and storage needs independently, without substantially disrupting the operations of their Redshift data warehouse.
To perform an elastic resizing across different node types in Redshift, follow the steps below:
- Locate and click on the cluster you want to resize in the Amazon Redshift > Clusters screen.
- In the cluster dashboard, click on Actions > Resize.
- On the following screen, you can choose from classic and elastic resizing. Select elastic resizing, noting Redshift’s estimate for the cluster downtime.
- Under “New cluster configuration,” make the desired changes to the number and/or type of nodes. You can accept Redshift’s recommended configuration based on your usage patterns, or select a custom configuration.
- Select the time at which you want to perform the elastic resizing. There are three options available: “Resize the cluster now,” “Schedule resize at a later time,” and “Schedule recurring resize events.”
- If you decide to schedule the resize for later, give the resize action a name and select the date and time at which it will be performed.
- Double-check the settings and click on “Resize now” or “Schedule resize,” depending on which option you’ve selected.
Elastic Node Resizing in Practice
We’ve discussed the new phenomenon of elastic resizing across node types, but how does it actually play out in practice?
Amazon Web Services cloud architect Bhuvanesh R. recently gave Redshift’s new feature a test drive—and the results were decidedly mixed. In his experiment, he resized 16 terabytes of data, moving them from an 8-node cluster of DS2.xlarge nodes to a 2-node cluster of DS2.8xlarge nodes.
From beginning to end, the new elastic resizing feature took roughly 25 minutes to complete. Although he praised the convenience and time savings that elastic resizing affords users, the author also mentioned a few noteworthy issues with the new feature:
- Whereas the Redshift dashboard predicted that the resizing operation would take between 10 and 15 minutes, it actually lasted 25 minutes, according to the author. As mentioned above, the original cluster remains read-only until the process is complete. Given this discrepancy, it would be a good idea for users to block off a generous maintenance window (just in case).
- Once the resizing is finished, users appear to no longer have access to the original cluster’s monitoring metrics and database statistics. If you want to maintain access to this information after the resizing, you should likely back up or export the data to an external location.
- Similarly, users seem to lose access to many of the Redshift system tables and views after the resizing. As the author notes, this makes it more challenging to compare the performance of the cluster before and after the resizing.
With these caveats in mind, however, elastic resizing across node types in Redshift appears to be a valuable new addition to the Redshift feature set. This is especially true if you plan to use elastic resizing on a regular basis. As of November 2019, Redshift supports automated elastic resizing using a scheduler feature—for example, reducing a cluster’s workload overnight or on a particular day of the week.
What have we learned from Amazon’s announcement of elastic resizing across node types in Redshift?
- Elastic resizing for the number of nodes in a cluster was already a valuable addition to the Redshift feature set, and a competitive alternative to Redshift resizing options such as classic resizing and snapshots.
- The choice of elastic resizing is likely best if you prioritize speed and convenience, and you don’t care if the cluster is unavailable during the operation.
- The elastic resizing feature has only been enhanced with the recent announcement that Redshift now supports elastic resizing for node types as well. Being able to change node types on the fly gives Redshift users more flexibility, scalability, and accountability for how they spend their AWS budgets.
- Although elastic resizing should complete within a matter of minutes, the speed of the operation in practice will depend on factors such as the existing workload on the original cluster, as well as the number and size of the database tables that will be replicated.
- Performing an elastic resizing operation across node types appears to delete information such as the original cluster’s monitoring metrics and database statistics. Be sure to back up this data if you want to retain access after the resize.
Here at Intermix.io, we share this passion for getting the most out of your Amazon Redshift deployment, making it both less expensive and easier to use. That’s why we’ve built our industry-leading analytics platform to better understand what’s going on in your Redshift data warehouse. Sign up today for a free trial of the Intermix platform, and start optimizing your use of Amazon Redshift now.
Join 11,000 of your peers.
Subscribe to our newsletter SF Data.
People at Facebook, Amazon and Uber read it every week.
Every Monday morning we'll send you a roundup of the best content from intermix.io and around the web. Make sure you're ready for the week! See all issues.