Like many companies that use the cloud, Wealthfront uses AWS Lambda. And while our lambdas look pristine in the web console and perform their highly specific duties well, there’s quite a bit that happens before our lambdas get to that state–I’m referring to CI/CD. However, depending on the given lambda, the processes to deploy its code varied drastically, and each had their own flaws. This blog post discusses the challenges we encountered with our existing lambda deployment patterns and how we resolved them by standardizing and improving the way we deploy them.

Challenges

Challenge 1: Too many deployment patterns

At a company of our size (about 150 engineers), there would ideally be one standardized way of handling lambdas, but we identified the following varieties in our deployment patterns:

• Number of repositories involved to make and deploy a code change: 1 or 2.
• Deployment workflows: Waiting around (no action needed), changing a hard coded image tag, or rerunning a Jenkins pipeline.

One of the principles of our engineering culture is Understandability, and having this many patterns was one potential sign that there was room to improve simplicity and uniformity. 

Challenge 2: Long deployment (and rollback) times

Additionally, getting a new change into production ranged from painless to painful. The main pain points were twofold: the number of pull requests required (up to three!) and the amount of time waiting around for code builds and the respective infrastructure-as-code (IaC) builds. At Wealthfront, we use IaC frameworks like Terraform, Serverless, and CloudFormation to manage cloud infrastructure.

Reverting a bad change was lower bounded by the runtime of the respective IaC framework and upper bounded by a combination of the previous and the runtime of building the code (i.e. Maven). In hard numbers, this ended up ranging from 5-30 minutes.

iac_build_time <= revert_time <= iac_build_time + maven_build_time

A Quick Primer on Lambda

Before we dive into our solution, there’s one API call in AWS Lambda to know about:

update-function-code

This API allows you to update the code running on your lambda. It takes in the name of the function you want to update and the location of the code artifact that you want to deploy on your lambda. New lambda invocations use the new code within seconds of making this API call.

Key Observation

As we explored the problem and solution space, we made one key observation:

Code deployment was tied to infrastructure deployment.

In order to deploy our code, we turned to Terraform, Serverless, and CloudFormation. But why? Not only do we rely on IaC to create the lambda resources themselves, we also rely on IaC to define what code to run on them. Under the hood, these frameworks will invoke update-function-code for us during a deployment. Unfortunately, despite the infrastructure itself not changing, we needed to wait at least a few minutes for IaC to run.

In light of this, we wondered: what if we just called update-function-code ourselves? Then, pushing out new code would no longer rely on an IaC build and would only rely on Maven to build our code. More importantly, rollbacks would only take seconds—not potentially tens of minutes.

Benefits

Acting on this observation could have several benefits to our infrastructure:

1. Code rollbacks could take seconds instead of 5-30 minutes.
2. New code changes would not depend on an IaC build and thus could be deployed faster.
3. New code changes would only require one pull request.
4. Onboarding engineers could be faster instead of requiring engineers to work with the unique interfaces of multiple IaC frameworks to deploy code changes.

Defining Requirements

These benefits are significant. But what would it take to be able to call update-function-code ourselves? Here was the most interesting requirement we faced:

Avoiding IaC State Drift

One problem that could have arisen from calling update-function-code independently of IaC was state drift. State drift refers to the state as defined in IaC drifting from the state in the actual environment. For example, here is a sample resource definition of a lambda in Terraform, the canonical IaC framework:

If we used any old image_uri in our deployments, state drift would occur. Alternatively, we could use Terraform-specific lifecycle rules to tell it to ignore changes to image_uri, but Serverless and CloudFormation don’t let us do that. Perhaps we could move around a prod or latest tag to our target image before calling update-function-code. Still, what if multiple lambdas use the same ECR repository?

Solution

To avoid IaC state drift, we required that the lambda IaC definition define its image URI using an image tag equal to the name of the function. For example:

Given these limitations, the final deployment workflow was as follows:

1. Move the image tag equal to the function name to the target image.
2. Call update-function-code using the target image.

This setup allowed us to leverage IaC without interfering with our external workflow and avoiding state drift. 

Implementation

With the high level workflow defined, we had to make it come to life.

A natural place to start was to evaluate the AWS managed solution: AWS CodeDeploy. It comes with a lot of things out of the box, like a nice user interface and a multitude of deployment strategies. However, we chose not to use CodeDeploy for the following reasons:

1. Moving an image tag as part of a deployment workflow was a highly specific operation, and CodeDeploy naturally did not support this out of the box.
2. Using CodeDeploy would introduce yet another deployment pattern that engineers would need to become familiar with.

Another principle of Wealthfront’s engineering culture is a preference to Build Over Buy. Our engineers are comfortable with reading and writing code to accomplish our specific needs. In fact, the deployment manager we use for our core microservices was built completely in-house. In the case of our lambda deployment workflow, we decided to implement it ourselves.

Wealthfront has “queries” (our internal API) that engineers use in their everyday workflows, microservice deployments included. This project was a great opportunity to reuse patterns: the deployment workflow described above was encapsulated in a query called DeployAwsLambda, and we used the AWS SDK to make the necessary API calls ourselves. Then, engineers could invoke these queries using a command line interface.

Conclusion

We had several deployment patterns for lambdas at Wealthfront, each of which had their own quirks. We made the key observation in our infrastructure that lambda code deployment was tied to infrastructure deployment. Decoupling the two allowed us to easily standardize the way we deploy code in addition to speeding up code deployments.

We didn’t overhaul or migrate away from any technologies (although we would love to consolidate them one day!); instead, we devised a solution compatible with multiple technologies to alleviate core pain points. We didn’t reduce the number of patterns end-to-end, but we did introduce common touch points to make the developer experience relatively consistent.

We strayed from the guidance that IaC should be the source of truth of one’s environment, and we implemented our own lambda deployment interface, adding to the number of moving parts. As with many engineering decisions, we found that the benefits exceeded the tradeoffs. If these types of problems sound interesting to you, perhaps consider joining Wealthfront!

---

Disclosures

The information contained in this communication is provided for general informational purposes only, and should not be construed as investment or tax advice. Nothing in this communication should be construed as a solicitation or offer, or recommendation, to buy or sell any security. Any links provided to other server sites are offered as a matter of convenience and are not intended to imply that Wealthfront Advisers or its affiliates endorses, sponsors, promotes and/or is affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.

All investing involves risk, including the possible loss of money you invest, and past performance does not guarantee future performance. Please see our Full Disclosure for important details.

Investment management and advisory services are provided by Wealthfront Advisers LLC (“Wealthfront Advisers”), an SEC-registered investment adviser, and brokerage related products, including the Cash Account, are provided by Wealthfront Brokerage LLC (“Wealthfront Brokerage”), a Member of FINRA/SIPC. Financial planning tools are provided by Wealthfront Software LLC (“Wealthfront Software”). 

Wealthfront Advisers, Wealthfront Brokerage, and Wealthfront Software are wholly-owned subsidiaries of Wealthfront Corporation.

Copyright 2025 Wealthfront Corporation. All rights reserved.