March 17, 2020
AWS Lambda has several limitations that make it unfit for some use-cases. In this blog we'll tackle the execution time limit of AWS Lambda, why is it a roadblock for some applications and what alternatives are there that offer an option to overcome this limitations.
The amount of time that Lambda allows a function to run before stopping it.
The timeout limit on Lambda used to be 5 minutes, but as it was a major barrier for some applications in using serverless, it was increased and since Oct 2018 Lambda can run up to 15 minutes.
On other serverless platforms, it varies from 9 minutes to 60 minutes maximum:
Azure Functions: 10 minutes by default, which can be upgraded to 10 minutes and even 60 minutes in their premium plan
IBM Cloud Functions:10 minutes
Google Cloud Function: 1 minute by default, which can be upgraded to 9 minutes
Many workloads are great candidates for serverless, and for them the execution time limit is more than sufficient. But, for some use case a 15 minutes timeout is a show stopper that would make the service owner to seek other solutions.
These use-cases are in general any long-running processes, such as video processing, big data analysis, bulk data transformation, batch event processing, very long synchronous request, and statistical computations.
For example, if you are working on a video processing app and you have a process that can potentially run longer than 15 minutes, AWS Lambda is probably not the best fit for your needs.
But what can be done?
In order to overcome the execution timeout, different organizations use different alternatives. Some of these alternatives are:
AWS Fargate is a serverless compute engine for containers. Fargate enables you to run containers without having to manage servers or clusters. As opposed to Lambda, Fargate doesn't have any time limitation and thus can be used as an alternative for Lambda in some use-cases.
In Rupak Ganguly's Blog on "How to use AWS Fargate and Lambda for long-running processes in a Serverless app", he demonstrated how AWS Fargate helps with processing a video file
AWS Fargate is set between EC2 and AWS Lambda on the spectrum and provides a solution to Lambda's limitations. While it's not a FaaS experience, it's still a serverless experience, so some still find it a good enough alternative to Lambda.
You can write Lambda functions that perform long-running tasks as recursive functions, as explained in Yan Cui's blog - "Write recursive AWS Lambda functions the right way".
However, as he explained in the blog, "as Lambda can recycle the container between recursions, it’s possible for you to lose the cached state from one invocation to another". Also, using this "hack" might have some problems in certain use-cases, for example when it's not possible to "break" some tasks to shorter processing units.
There are other serverless platforms that allow you to top AWS Lambda's execution time, such as:
Some claim that if your task is supposed to run for a long time, then lambda is not for you and you should probably consider using VMs or containers instead.
AWS Lambda is not the ideal solution for long running services, but that doesn't necessarily mean that FaaS is not a possible architecture for these workloads. Solutions and alternatives vary, but there's probably a serverless solution for your long running process.