Serverless solutions are cheaper… right?
When infinitesimal cost X infinite scalability = serious money
Serverless solutions can save money - especially the extra costs of ownership that come with having to look after infrastructure you don’t even need most of the time. A talented solution architect I've been working with just said he’d heard serverless can be up to 4x as expensive as running servers.
Serverless can be way more expensive than 4x.
Once upon a time….
I was the technology director for a famous author’s ecommerce site. We had ~800 rarely-changing images in S3 that users download often, so I asked the development team to set a cache header on each one to save our bandwidth costs.
The team, who were excellent, suggested putting CloudFront in front of S3, but I thought the knock-on effect of changing the download URL wasn't worth it. So they decided to use AWS Lambda, just for this one small thing.
Using Lambda for a tiny task like this is perfect - we’re just paying for when we run it, which was going to be hardly ever. We couldn’t even work out how cheap it was going to be.
Just one small thing
There is an event when an object is changed in S3 and you can invoke a Lambda that sets the cache header. One simple step, completed almost instantly. They tested it on Friday morning on a single S3 object - and it worked. Because it was a Friday, they didn't make the change for the other 800 objects. Good decision.
By Monday morning we’d spent $3,000 on setting one header attribute on one S3 object.
Wait… what?
When you set the cache header, the S3 object is considered updated, which fires the event, which runs the Lambda, which sets the header, around in a very fast loop that nobody can see. Your servers don’t slow down. Your website doesn’t crash. So you don’t really notice. Who would worry about wiring such a tiny piece of functionality into their observability stack?
If we had rolled out this change fully on Friday we’d have spent $2.4m that weekend - enough to endanger the company.
This is about 500 times what we were spending on servers annually, all of which had vast spare capacity for setting those headers.
When things run wild, run silently and you have ~infinite scalability, you can spend a lot of money, fast, even on incredibly small, cheap operations.
Happily ever after
Amazon hadn't documented that setting an S3 object header fires the object change event, so the team quite reasonably assumed the event was only for content updates. They challenged Amazon, which very kindly agreed to write the $3,000 off.
It could have been a painful, expensive and possibly terminal lesson, though.
Infinitesimal cost X infinite scalability = serious money.