I've been using AWS lambda a bit recently, mostly as a way to glue together various bits and pieces.
Maybe I'm doing this wrong but does anyone else find the experience to be really frustrating?
I can unit test bits of the code just fine, but at some point I always end up stuck in a slow feedback loop where I deploy the code, do some manual invoking, go and dig through the logs in CloudWatch, add another print statement in my lambda... and so on.
What I want is to run the lambdas locally, ideally more than one, and then exercise them with streams of test events (perhaps captured from a real environment). It would be quite cool if I could define BDD style tests around them too.
Anyone have any suggestions or share my frustrations?
I have heard localstack is quite good although I haven't given it a go yet. Would that work for me? I did try SAM but I was a bit underwhelmed and I don't want to use a separate IaC tool for these.
Alternatively, do other FaaS providers solve this problem?
Thanks for any help.
Programming in the 1960s to 80s was like this too. You'd develop some program in isolation, unable to properly run it. You "submit" it to the system, and it would be scheduled to run along with other workloads. You'd get a printout of the results back hours later, or even tomorrow. Rinse and repeat.
This work loop is incredibly inefficient, and was replaced by development that happened entirely locally on a workstation. This dramatically tightened the edit-compile-debug loop, down to seconds or at most minutes. Productivity skyrocketed, and most enterprises shifted the majority of their workload away from mainframes.
Now, in the 2020s, mainframes are back! They're just called "the cloud" now, but not much of their essential nature has changed other than the vendor name.
The cloud, just like mainframes:
- Does not provide all-local workstations. The only full-fidelity platform is the shared server.
- Is closed source. Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. You can't peer into their source code, it is all proprietary and even secret.
- Has a poor debugging experience. Shared platforms can't generally allow "invasive" debugging for security reasons. Their sheer size and complexity will mean that your visibility will always be limited. You'll never been able to get a stack trace that crosses into the internal calls of the platform services like S3 or Lambda. Contrast this with typical debugging where you can even trace into the OS kernel if you so choose.
- Are generally based on the "print the logs out" feedback mechanism, with all the usual issues of mainframes such as hours-long delays.
I feel like the service is leaving a bunch of easy work on the table that would let people have a local development process that mirrors the remote development reality.
Arguably that’s a more dignified form of programming for a more elegant time but it doesn’t really mesh with the reality of “try fail print retry”.
Otherwise, you get a mishmash of fantastic visibility into your workloads and better provisioning cycle times, which then immediately get cancelled out by developers who have to work 3x as hard to do the same thing they were doing before.
Unfortunately, what it most often seems to be used for is triggering side effects in response to some event, which is exactly the opposite of a pure function. Local testing of that logic is probably reliant on mocking all the other services it integrates with. Beyond that, the next step tends to be remote integration testing on the actual cloud hosting service, using some sort of separate staging infrastructure that will possibly be shared among many developers.
Some of the ideas and flexibility that should be offered by cloud hosting are appealing, but in practice, it often feels like we've regressed decades in our software development practices, discarding successful ideas like shorter feedback loops and unit testing and one-step build/deploy processes.
Ideally, they should first come up with a spec - and provide reference implementation.
Every provider will implement solutions for these standard specs. That way customers can use services for the spec. Customers can move between providers or work with multiple providers at the same time. Level playing field & let the best provider win the customer loyalty.
But, all providers want to lock in customers. So, we are at the wrong end of the stick.
> Does not provide all-local workstations. - Solutions like minikube fix this.
> Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP.
Kubernetes has the same developer experience everywhere. As far as OP's serverless trouble, why not just use knative?
(I haven't tried it yet as the whole functions fad still hasn't hit me)
See also “big data” and machine learning.
I'm a big fan of [OpenStack](https://www.openstack.org/) as it is open source and definitely enterprise-grade.
https://www.devopsparadox.com/episodes/does-history-repeat-i...
Do you hold out hope for a similar transition to a better local development workflow? What do you think it might look like?
Openness, standardisation, commoditisation, and low-tier equivalent offerings are an anathema to the vision of these mega-corps. They will fight tooth and nail to prevent it.
Look at what Microsoft is doing: Instead of moving Azure features down to their IDEs, they are moving their IDE into the cloud! VS Code is Electron-based, which is to say it is just a local web server plus a web browser shell. It's not rocket science to move the server component into the cloud, and have the standard customer web browser be the front-end. I've noticed recently that this has started to happen. There's already a SQL query editor based on VS Code, and there's editors popping up for App Service and I think Functions also (equivalent to AWS Lambda).
This is the ultimate lock-in, where you develop "on" the platform directly, and have no hope of ever reproducing any part of it locally: the build system, the debugger, source control, tracing tools, or the IDE itself.
Other people in this thread mentioned local Kubernetes development flows. They're too busy drinking the Kool-aid. The cloud vendors are not interested in ever making this a long-term viable option and will find creative ways of making such open development practices non-tenable.
I envy the geniuses at Microsoft who can convince 60k+ software companies like mine where we're all engineer or mathematicians, led by incredibly clueless IT decision makers :D I've talked face to face with people from an other continent in charge of our switch to Azure, and he was neither aware nor involved in even the decision of Azure vs AWS vs Google Cloud. But he was in charge...
I came to learn that the main reason we chose Azure (but I think the biggest problem was to choose Cloud) was because we already had teams doing Windows maintenance, server management, etc: we had way more relationship with MS already than with Amazon or Google. Now we have it, it's beautiful, nobody uses it, people grudgingly do trainings, and make big presentation about Cloud strategies, while waiting for legal and compliance to allow us to connect to it. Since we figured after the contract was signed that it would be mostly illegal for us to give Microsoft any of our client data...
Whatever, happy to see sometimes other people struggling with the bewildering fact that we somehow regressed :D
> You'd get a printout of the results back hours later.
Not sure what you are trying to say: What you describe is not time-share, but batch processing.
When it comes to Lambdas, given the reasons above, there's only one thing that can improve the experience: PROXY. Before i went on parental leave i had the idea is creating a proxy lambda which can be configured with an IP and port number. That IP and port is for your local dev environment. This way, when developing you can instruct a live system to short-circuit and to proxy calls to a local lambda available om the respective port. Trigger end-to-end tests by invoking AWS services that will eventually call the proxy lambda, and then your local lambda with the same environment/context/input, reply with output which will reach the proxy lambda, which will output the same content forward.
That's pretty neat if so, I might have to try that out. Are there any downsides to this approach do you think?
Thanks for your thoughts on emulators/tests too.
You can use this in your local dev env and even in integration tests. I don't need to do this a lot but sometimes it helps when troubleshooting.
Also, you might actually be building a tool which can be used from the command line. I recently took an existing command line tool (written in Go) and ported it to Lambda to expose it as a chatbot.
So - reverse engineer and monitor in-transit behavior it is...
How do you automate this?
Back to the point: reality is that many build their AWS environment (prod) manually, maybe they duplicate once (dev) also manually, maybe they use some automation for their "code" (lambda) but that's it. This implies it's practically impossible to run end-to-end tests. You can't do that in prod for obvious reasons and you can't do it in dev either - you have many devs queueing, maybe dev is not in sync with prod etc.
My team ran cloudformation end-to-end. We actually orchestrated and wrapped cloudformation (this is yet another topic for not using terraform etc) so that if smth couldn't be done in CFN, it would still be automated and reproducible. Long story short, in 30 minutes (it was this long because we had to wait for cloudfront etc) we had a new environment, ready to play with. A total sandbox. Every dev had their own and it was easy to deploy from a release artifact or a git branch to this environment. Similarly you could create a separate env for more elaborate changes to the architecture. And test in a live environment.
Finally to your question: how do you test end-to-end?
If we talk about lambdas because that's where the business logic lies in a "serverless" architecture, then the answer is by calling the system which will eventually call your lambda/s along the way. If your lambda ia sitting behind AWS gateway, then fire an http request. Is it triggered when objects land on S3? Then push some object to S3. How do you assert? Just the same - http response, S3 changes etc. Not to mention you can also check cloudwatch for specific log entries (though they are not instant).
With this type of a setup, which sounds complex, but it is not since it is 100% reproducible (also from project to project - I had several), adding this proxy-to-my-dev-machine lambda would mean I can make local changes and then fire unit AND end-to-end tests without any changes pushed to AWS, which is the main time/energy consumer imo.
PS: sorry for the wall of text. Like i said i recently realized that the development realities have huge discrepancies, so i tried to summarize my reality :)
do you have a git repo for your proxy code?
I only had a PoC and then told myself this would be brilliant as an abstraction
I find it more simplistic that Digital Ocean , tbh
The purpose of the AWS Console, AFAICT, is:
(1) To be used by semi-technical ops and management types, and people doing tutorials, and
(2) To drive everyone else as quickly as possible to the APIs/CLIs/SDKs.
If I'm honest, I do find AWS in general fairly high friction at times but it's way better than renting servers directly I think.
It's the only cloud provider I've really worked with to be honest though (other than a few experiments), maybe some others are much better in this regard?
I don't really get where you're coming from on renting servers directly, using AWS is far more expensive if you actually need a server for a while.
Though I do admit being able to fire up a server for a couple of hours and then delete it is pretty awesome.
I’ve been thinking a lot about AWS lately and it definitely seems to me like a modern take on what IBM did in the 80’s.
In my own work with serverless I think about this a lot. In my day job, we built a new service and we pay AWS a few dollars per month to run. It would have cost us around $100 a month in AWS costs. However, the operational complexity is extremely high and we are also coupled to another service via Kinesis. For a billion dollar business, the trade off doesn't seem worth it.
I wonder how much of serverless is just AWS firing at Google (and Heroku, DO, etc) and other competitors. It certainly hasn't made my life as a developer easier. It's certainly much cheaper for some use cases but the complexity of the system goes up very quickly, and you're end up having to manage a lot of that complexity using an inferior tool like Cloud Formation (or Terraform).
It depends on what you mean by "serverless". Serverless webapps? Sure, maybe a fair bit. But the real killer feature of the Lambda service is that it's a universal extension mechanism for other AWS services. Want to run some shoddy virus scanning whenever an object is added to a bucket for compliance reasons? That's a lambda. Want to manipulate Firehose records before they are sent to the sink? That's a lambda. Want to execute some crazy Python function from an Athena SQL query? You guessed it. That's a lambda.
So when you say "It certainly hasn't made my life as a developer easier" it means you haven't needed to do these bespoke gluing together of stuff, because "being able to do this somehow" is infinitely easier than "not being able to do it at all because AWS hasn't exposed an API for this use case".
I think the true geniuses are the ones who spent so much time trying to sell people on "Server Admin is hard, lets hire it out!" as well as building the billing and metering systems.
In a gold rush, screw panning, sell shovels (and pans).
Appropriate for a small amount of apps, but used inappropriately by a lot more for a while because it's the hot new thing.
The complexity bogeyman is a red herring. What is software development if not a constant fight between increasing complexity and adding features, including making systems more robust and resilient and reliable?
Sure, you are free to claim that function-as-a-service paradigm is very complex because the code runs somewhere else, and you have to use a separate workflow than the one you're used to, and that you actually have to write unit tests to have some assurances that stuff does work as expected.
But what would be the alternative approach?
Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.
AWS Lambda is a fancy way to let developers develop event-driven systems by just writing tiny event handlers and plug in events. It's no rocket science. In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt. I'm terms of web services, they are a service where you just have to write the request controller bit after that runs after all auth things passed and request messages were parsed and validated. Considering this, isn't it unreasonable to talk about increasing complexity, when everything was made.far simpler than what it would otherwise be?
It's not really a red herring when Lambda mostly forces communication between different application layers to be over the network. You suddenly have to deal with rate limiting, retries, bulkheading, etc. in a lot more places than you would even with even a SOA.
> you actually have to write unit tests
I'm not really sure unit tests are a differing factor in testing functions as a service and other software. The real issue with testing is that it's very difficult if not impossible to do more than unit testing locally. You're often using docker containers or mock services to try and simulate what is going on in production which is insufficient. Things like CloudFormation can't be done locally (and this can be where a lot of gross complexity lies), so ultimately you have to set up an identical-to-prod testing environment and do the bulk of your testing there (which you probably should have anyway, but that's besides the point). Things like localstack HAVING to exist but not being supplied by AWS is a symptom of something gone wrong.
> Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.
Why is this obviously no? It isn't to me. You likely need a lot less these type of things if you're running a monolith.
> In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt.
It seems to me you're ignoring all the extra stuff you HAVE to have to deploy, secure, and maintain serverless. There is a lot more than the code for the function themselves.
I often see ppl adopt one of two development patterns:
1. Locally mock all the services that your Lambda function uses. Like API Gateway, SNS, SQS, etc. This is hard to do. If you are using a tool that mocks a specific service (like API Gateway), you won't be able to test a Lambda that's invoked by a different service (like SNS). On the other hand a service like LocalStack, that tries to mock a whole suite of services, is slow and the mocked services can be out of date.
2. Or, you'll need to deploy your changes to test them. Each deployment can take at least a minute. And repeatedly deploying to test a change really slows down the feedback loop.
SST lets you develop your Lambda functions locally while connecting to others AWS services without needing to mock them locally. Behind the scene, the lambda requests are streamed to ur local, runs ur local function, response streamed back to Lambda. So you can iterate on ur function code without redeploying.
Think of your Lambda functions are Live Reloaded. You can see it in action here — https://youtu.be/hnTSTm5n11g
I’m the core maintainer of the project. And folks in our community are saying using SST feels like "working in the future" ;)
I'd love for you to give it a try. We are also a part of the YC W21.
Also, this guide they put together is definitive: https://serverless-stack.com/#guide
Had I found that years ago, I would have saved SO MANY headaches. I've implemented local debugging (hell, and doesn't actually replicate AWS). I've followed the CloudWatch logs, just like you, painful.
The SST local debugging with lambda is the best way forward. Deploying micro services on seed.run is also the way forward.
I'll check it out!
You can also direct any questions/feedback to frank@anoma.ly :)
I run my lambdas locally with a single command: serverless offline
If the lambda has an http endpoint, it creates the endpoint at localhost/<endpoint> and I'm good to go, it even does live reloading as I edit my code.
If the lambda runs off AWS events, I can invoke the lambda locally with a command, and point it to a JSON file of a simulated AWS event. I get my local rails app to create these AWS event JSON files, so that I can test end to end locally. Works well for my purposes.
To deploy I just run: serverless deploy --stage production
Which sets up all the necessary additional services like API Gateway, cloudwatch etc.
I can't imagine using AWS lambda any other way.
AWS provides a docker image for emulating DynamoDB which works great for local dev and will commonly be paired with lambdas.
Another option I have used recently for implementing node web services is https://github.com/apex/up which also has a nice local dev experience.
BUT .. I should also say that while I started ~5 years ago with serverless framework, in the past year I've completely transitioned everything over to the Azure Durable Serverless Functions.
AZ has their own same-same-but-better "Core Functions Power Tools", it's got a lovely & zippy cli interface built in node that is separate/decoupled from the "az cli". Just like serverless you can run/dev/test locally, then quickly deploy to azure. The debugging integration with VS code, distributed tracing, even "AZ Remote Debugging" with "Azure Hybrid Connections" creates super fast inner/outer development loops.
If you like the "serverless model" then you also need to check-out AZ "Logic Apps". Logic custom connectors totally kick ass imho since they provide a ton of insight into how data is moving through the application, basically you deploy and you get instantaneous telemetry.
Deploying software that costs nothing (except negligible storage fees) unless you're using it AND would require an ELE (i.e. meteor strike) to break is very cool.
It was surprisingly hard - the local development of lambdas is still very raw, documentation is scarce, and there are various small issues appearing here and there. I should probably write a blogpost about how to setup decent developer environment for AWS CDK and Lambdas, because there's not much on the Internet about it.
I set up the whole AWS infrastructure via AWS CDK. I have one TypeScript file, that creates a stack with lambdas, dynamodb tables, API gateways, S3 buckets, wires up the secrets manager, etc - all of that in 2 environments - dev and prod.
AWS CLI also can generate a Cloudformation YAML file from the CDK file (via `cdk synth`), which could be fed into SAM. So, I generate a `template.yaml` Cloudformation file this way, then run SAM like `sam local start-api`, and it runs exactly the same lambda locally, as in AWS, using that Cloudformation YAML file. SAM also supports live reload, so if you change any source files, it will automatically get those changes.
So, okay-ish developer experience for lambdas is possible. There're caveats though:
* I couldn't figure out how to use "nameless" Dynamodb tables (i.e. when CDK assigns the name to them automatically), because if I omit a table name in CDK template, then the local lambda and lambda in AWS assume different names for some reason.
* Locally, the binary outputs don't work. It ignores `isBase64Encoded` by some reason, and lambda just returns Base64 output, instead of binary.
* And the main problem - local lambdas are SLOW. It seems like it restarts some container or something under the hood on each API call, which adds 2-3 seconds to each API call. So, the calls that should be like 30ms, are actually 2 seconds now. This is super frustrating.
Each use case is so unique, that it is likely difficult for them to account for every single situation where Lambdas are being used. Instead they opt for a generalized and non-specific documentation site, which results in docs I find practically useless.
I frequently find solutions to problems for AWS on someone's obscure blog, who after many days of suffering stumbled onto the answer, rather than finding anything of use directly from those who wrote the software.
Does a site exist where programmers can submit useful findings? Similar to stackoverflow, but rather than asking for questions, you just submit answers. I would think being able to search across such a crowdsourced utility would save programmers a ton of time.
SO kind of lets you do this. You can write a question and immediately answer it yourself - there's an option in the question UI for it.
But they're too limiting. Cloudflare Worker should be a pure JS script (that looks like a service worker script in a browser). You cannot use any Node libraries. In my app I need to generate images on the fly, so I couldn't figure out how to do that in a Cloudflare worker, so I ended up creating lambdas for those, and proxying the calls from Cloudflare worker to those lambdas.
KV (their key-value store) is way too simple. No secondary indexes, no sorting, no querying (you can only sorta query list of keys, but then you have to fetch a record by each key individually). My app hit the limits of KV very quickly.
Also, there's no monitoring or any good way to do backups for KV. So, it's almost unusable in a real-world production system, except as maybe a cache layer.
Monitoring was a big pain for Cloudflare Workers. Like, even accessing production logs is weird - you have to "tail" them from your CLI. Even Cloudwatch is 100x better here.
Development environment is okay, I guess, but the local server crashes constantly. Also, it kinda forces you to use Webpack and bundle the sources. It also doesn't have anything like layers in AWS lambdas, and they have a limit of 1MB for a bundle, which I almost hit (I had like a 900kb JS file), so that part got me worried too.
Basically all of that made me worry I will hit a wall soon with Cloudflare Workers, so I moved to AWS. It's not piece of cake too, but at least I get a way to run proper Node scripts, a decent document database, continuous backups and pretty good monitoring out of the box.
Cosmotic's comment kind of confirms my suspicions to be honest.
I posted elsewhere in this thread but checkout SST (https://github.com/serverless-stack/serverless-stack), it lets’ you work on your Lambda functions live without having to mock AWS services locally. Here’s a short demo (https://youtu.be/hnTSTm5n11g)
Literally today I had a billing problem with AWS and it was one of the worst, broken UI experiences I've ever seen. Involving money to boot (usually those things have higher standards).
They are really bad at quite a lot of things, it is what it is - there might be some rhyme or reason to it, for example, they might not care about documentation, or may be much more focused on bread and butter ec2, bit it's totally plausible that there are a lot of frayed edges.
You know that difficult-to-attribute quote: 'Don't attribute to malice that which can just as easily be attributed to incompetence'.
Here’s a short demo (https://youtu.be/hnTSTm5n11g)
I'd love for you to give it a try.
However, in response to your specific pain point - after initializing a new functions project you can immediately run an 'npm run serve' and have local execution firing for any scaffolded endpoints. These - by default - will connect to a project associated RTDB/Firestore instance without the need for additional service account/IAM configuration.
I've always enjoyed that this was a part of the mainline dev flow - and it removes a huge barrier of entry for me (mostly mental) to spin up little toy projects or test endpoints whenever the thought crosses my mind.
Note that this library only supports scaffolding one function at a time per process - so you'd have to create a separate process [and use a separate port] for each function you're using. (This includes event-based functions - everything is HTTP under the hood, and this library is no different.)
2) Deployment time - you're definitely not the first to bring this up. :) We're working internally on improving deployment times.
(Disclosure - per my username, I work at GCP.)
[1] https://cloud.google.com/functions/docs/running/function-fra... [2] https://www.npmjs.com/package/@google-cloud/functions-framew...
Cloudflare's Workers have a good dev experience. It takes a second to build and locally preview a function. It live-reloads when you make a change. Later, it takes a second to build and deploy. On the flip side, Workers can't run arbitrary Docker containers. Your code has to be JS/TS or WebAssembly.
I'm not affiliated with Cloudflare, just a satisfied customer.
My experience has been the opposite.
Wrangler doesn't give you any meaningful errors (since it converts JS errors to Rust) and it crashes way too often.
Recently I worked on a Workers project and I had an undefined missing variable. The only error Wrangler would output is "error".
IMO Workers are only viable for really small and simple projects.
Containers are coming to the Cloudflare edge: https://blog.cloudflare.com/containers-on-the-edge/ (inevitable really)
Nit: Workers should really be compared to Lambda@Edge instead.
Am I right in saying that they serve a somewhat narrower use case than lambdas though? My understanding is that they are mostly used for things like modifying requests/responses rather than for building a whole backend.
As you say, it only does the 'host lambdas locally' bit though. If you wanted to do stuff like test the interactions between lambdas, or test them over HTTP (for lambdas that act as components in REST APIs), you'd need to build something yourself.
You’re not testing that HTTP works, you’re testing that the two functions work together. If you can abstract a calling interface between them with different backends you’ll have a better time developing lambdas, while also not being tightly coupled to them.
Create an RPC container object that has method stubs for the other lambdas you want to call. When you deploy it on AWS configure it to make an HTTP call; when running it locally, configure it to just call the code as normal.
Just think of the HTTP call and JSON blob as the calling convention. The API is really simple.
I think one of the biggest mistakes people make when developing Lambadas is designing them differently to normal software.
AWS has Step Functions, which is a vaguely functional and declarative language for linking your lambdas together. It’s a bit verbose and annoying to write, but at least a it makes your lambada setup repeatable.
The state machine which actually runs your step functions is publicly available.
https://docs.aws.amazon.com/step-functions/latest/dg/sfn-loc...
What people need to understand about testing against AWS is that the SDKs provides the typing guarantee. So it’s nearly impossible to make an invalid call with any AWS SDK.
Moreover, doing a test along the lines of ‘post to S3, then get the object to verify it has been posted’ is pointless too. Because that’s not how AWS works. In a real application, you don’t have any guarantees the object exists in S3 immediately afterwards, just a high probability that it will become consistent relatively quickly. If you do such a thing in a real world environment, you’re liable to have intermittent failures.
The eventually consistent behaviour is more obvious with other APIs like the EC2 interface, where no one expects an instance to be created immediately.
There are a few exceptions to this rule, but they’re not accurately modelled by localstack.
What people need to test when developing AWS applications is that there _was_ a call to an AWS API, not that the AWS API behaved with all of its documented side effects. You can’t unit test the black box of AWS, you have to QA it as part of an application release.
In principle I try to avoid Amazon Web Services that lock me into the platform. So, for my dev stack I run a few other containers to give me the equivalent of the prod environment, RDS (MySQL or PostgreSQL), ES (OpenDistro), S3 (Minio), and SES https://github.com/mozey/aws-local
Dev is easy because I can test everything locally, and the resulting code should be portable to any other cloud.
Obviously this approach might not be feasible if you choose to use AWS that do not have equivalent self-hosted options.
[1]: https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm... [2]: https://github.com/aws/aws-lambda-runtime-interface-emulator
I didn't join the dots up for this use case though, thanks!
P.S you can even attach the VS debugger to the live, deployed remote function and get full debugging (although quite slow). Don’t forget to deploy with the Debug (rather than Release) profile if you wanna do that. Breakpoints and all!
One caveat with these options is that the Function won't get a Managed Service Identity since it's running locally, so if you have code that relies on an MSI you'll have to create a Service Principal and make your code support SPs too.
Apart from that, the beahvior of the Function running locally or in the cloud should be identical.
[1]: https://marketplace.visualstudio.com/items?itemName=ms-azure...
[2]: https://github.com/Azure/azure-functions-core-tools/releases Azure.Functions.Cli.linux-*
It would be great if there was a consistent way to do all of them.
https://github.com/jorgebastida/awslogs
It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)
Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.
Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.
1. If you are building APIs and using Lambda functions as targets from an API Gateway API, look into libraries like serverless-wsgi (Python) or wai-handler-hal (Haskell) that translate between API Gateway request/response payloads and some kind of ecosystem-native representation. Then as long as you're writing code where all state gets persisted outside of the request/response cycle, you can develop locally as if you were writing for a more normal deploy environment.
2. Look into the lambda runtime interface emulator ( https://github.com/aws/aws-lambda-runtime-interface-emulator... ). This lets you send invoke requests to a fake listener and locally test the lambda more easily. While the emulator is provided in the AWS container base images, you don't need to run it inside a container if you're deploying with zip files. (AWS-provided container images automatically enable the emulator if not running in a lambda runtime environment, and using docker for port remapping, which is nice but not at all required.)
3. Get really good at capturing all requests to external services, and mocking them out for local testing. Whether this is done with free monads, effect systems, or by routing everything through gateway classes will depend on your language and library choices.
It would be quite a powerful workflow for running integration tests. I shall do some research!
We had to detect if running locally or deployed in our code because Cognito auth was giving us trouble when running locally, and Step Functions can only be tested deployed.
Cloud Watch logs take several minutes to appear, funcs take some time to package and deploy (even with caching) - making feedback loop very slow.
I personally think the best way is to develop serverless funcs on platform - using the built in editor. You get logs instantly. But can't get devs to give up their IDEs...
While you should take my comment with a grain of salt, for me it is a live testament that the serverless development paradigm works.
That being said, I also believe that it depends on how exactly you work and develop products/features. That is probably more important than the technological approach used. We have some very strict mental model about lambda which do not quite fit the mental model used for microservices.
But if your function more or less resembles a little web app that maybe talks to S3 and DynamoDB, this works really well.
Curious what sort of errors OP is running into that cause friction.
I posted elsewhere in this thread but checkout SST (https://github.com/serverless-stack/serverless-stack), it lets’ you work on your Lambda functions live.
Think of it like your Lambda functions are Live Reloaded as you make changes to your code. You don't need to mock AWS services locally, and you don't have to wait for CloudFormation to deploy your changes.
Here’s a short demo (https://youtu.be/hnTSTm5n11g)
- cloud native is weird/hard - old ways were better! - misconceptions about boundaries and aspects
Remember folks. When choosing technology, your selections always depend on a list of concerns, including:
- cost - complexity - developer audience - support audience - change management - vendor relations - mapping out dev strategies - getting consensus from team
No matter what tech is used, you still need to go through planning and design.
More often than not, cloud enables us to hack first. That is never a good strategy.
There’s nothing inherently bad about cloud native. It’s just different and _can_ be highly beneficial to outcomes.
I was in that position once, with Microsoft functions, which my boss wanted to use, but fortunately convinced my boss to get rid of it. Maybe others don't have the same luck.
And, why is this such a wild expectation in 2021? Is it that so much to ask?
It uses AWS Lambda underneath but it makes the dev experience just awesome.
I'm using python mostly atm. I've played around with go/node too (both of which I'm familiar with in and of themselves) and I was thinking about switching over to one of those already.
The dependency management is easier for both of them than with python I found.
I don't know why you need to deploy to test Lambda code; you can hit remote AWS stuff from local. The AWS SDK picks up your local config's IAM role the same way as it picks up the one granted to the Lambda itself. You don't need localstack for this, just an account in your AWS organization with the right role attached.
Packaging dependencies was a little weird to figure out, but the docs [0] are very good. A simple shell script can do the packaging work; its just a few lines to make the zip file.
[0]: https://docs.aws.amazon.com/lambda/latest/dg/python-package-...
We have containers which unify local environment and production environment. In my opinion (and experience), there aren't any more efficient shared mediums to work with.
‘aws lambda invoke -e myevent.JSON’ goes a long ways, because you get the full response object back.
I worked on Netlify Dev, which does offer a simulated lambda environment for testing locally: https://news.ycombinator.com/item?id=19615546
I was also quite disappointed to see that AWS Amplify did not offer a running dev server for lambdas, best you could do was a local onetime manual invoke and even that often failed (mismatched to the live lambda environment where it succeeds)
I lead Developer Advocacy for Serverless at AWS and we're constantly looking at ways to help streamline and simplify this process for you. You can find a lot of information on a site we've launched to rollup lots of resources: https://serverlessland.com .
Right now the best dev process is a mix of using local emulation via AWS SAM CLI (http://s12d.com/sam) or tools like localstack. But it's true that you'll never be able to completely mimic all the various services and their functionality.
Would love to know more of what left you feeling underwhelmed by SAM. It has almost all the functionality of CloudFormation, which is quite a lot.
Thanks, - Chris Munns - Lead of Serverless DA @ AWS
We recently started using AWS Athena and I was shocked to find no local test environment is offered by Amazon. Eventually I built a test container that uses Hive and Presto and allows my integration tests to deploy their test data to the Hive filesystem for Presto to access it. Unfortunately, Presto deviates from Athena so I can only approach the production environment during development, which leads to much lower confidence than we would have had otherwise.
Essentially, after all the tests are green, I'm still not sure if my code is production ready. Which is bewildering to me, we made all this progress in the past decade with continuous deployment, this just seems like a large step back.
It's not like it's that hard to fix for Amazon, they could at least offer local test containers for their cloud offerings. They just choose not to.
There's also a docker container to mock dynamodb locally.
When I'm just working logic... Unit tests are best, but you could also keep the lambda invitation endpoint to lambda_handler, then write your own main() function to do some setup (like assume role to the same role as the lambda execution role, set up env bars)... Then end by invoking your lambda_handler locally... Basically main() is your pathway to running the lambda locally.. with only the minimum code required to "fake" being an AWS lambda.
1: https://docs.aws.amazon.com/serverless-application-model/lat...
It's just so slow to debug and develop on AWS Lambda
Pre-packaged lambda functions are a pain to deal with if you're standardizing on something like terraform for all AWS infra provisioning - you need to jump through extra hoops to repackage the lambda functions into something you can manage.
Replacing the zip deploys with Docker images is a good first step. I just wish they'd support inter-account ECR -> Lambda usage - ATM you still need to replicate the images to each account separately.
Over the years of building serverless applications there are some unique differences in building them compared to traditional applications you build entirely locally then "throw over the wall" to be run on a server somewhere. And for this reason there are many teams and organisations working to make the developer experience better, including making deployments to the cloud to test easier and faster. Unlike mainframes of old, new code can be deployed in seconds and logs available in less than that when testing invocations of your Lambda functions. For example, the Serverless Framework has a command to deploy just the code changes of a Lambda and does so in seconds. There are other tools that do similar.
Its a really broad topic to get into but am more than happy to do so if you wish or have any specific questions I can help you answer. If its easier to ask privately feel free to do so
And I would just like to leave saying that I am a proponent of serverless development not because I happen to work for a company bearing the name, but because I sincerely think its the future. They just happened to spot that and asked me to join them.
For debugging in the cloud, besides cloudwatch you can use x-ray and have a full picture of your services and segments.
And if this ins't enough and you really need to test in the cloud, then use CLI or serverless to upload your lambda(few seconds tops)
invooke the lambda with test data via cli https://docs.aws.amazon.com/cli/latest/reference/lambda/invo...
and tail the logs via AWS CLI.
https://awscli.amazonaws.com/v2/documentation/api/latest/ref...
I mean, those steps aren't much different to running tests locally, just create a script for it =)
1. Some great tooling out there to shorten the dev loop on k8s. I use tools like skaffold or apache kamel k, but telepresence looks interesting too. 2. Minikube solves the "Localstack" issue. It's an official project that runs the exact infra, not just a replica of the APIs. 3. logging is far more straightforward than cloud watch. 4. It runs the same everywhere.
>https://bref.sh/ -- it allows running PHP apps on AWS Lambda effortlessly. With all the serverless craze, it's a very important piece of the ecosystem. After writing PHP apps for two decades, I am absolutely stunned how easy it is. Amazon blogged about it at https://aws.amazon.com/blogs/compute/the-serverless-lamp-sta...
See the following for more information:
https://vercel.com/docs/serverless-functions/introduction#lo...
I tried a few solutions earlier in the year for a side project (vercel, netlify functions, Cloudflare workers, begin) and found vercel to be the outstanding choice for developer experience, features (wildcard DNS for example is fully supported on their free plan) and support.
I feel as though there are similar amounts of friction across AWS offerings that make them unpleasant to work with in general, too. I'm numb to it now, but imagine my surprise when I actually went to use it for the first time after being hyped up about AWS.
For these reasons, and because you pay a significant premium for network traffic on AWS, I never use AWS for personal projects and I'm very happy with that choice.
With Google Cloud Functions, you can run your function locally using the Function Framework libraries (available in all Cloud Function languages) [0]. These libraries are used in production Cloud Functions, so you're not emulating the runtime. The feedback loop / devX is pretty simple with just 1 command to run the function locally like a library for every language.
We have some guides on local development here [1]. Our samples have system tests that use the FF.
You also can bundle your function in a container after building it with `pack build` [2]. This provides an exact container environment with things like environment variables and system packages.
We're working on improving eventing devX, but we have some `curl` examples here [3]. Connecting multiple functions together eloquently is still something to be desired though.
[0] https://github.com/GoogleCloudPlatform/functions-framework [1] https://cloud.google.com/functions/docs/running/overview [2] https://cloud.google.com/functions/docs/building/pack [3] https://cloud.google.com/functions/docs/running/calling
localstack
mintel/pytest-localstack
charlieparkes/boto3-fixtures
Instead what I got was a (massively over engineered?) front end (the API Gateway) where the default is to add 10s or 100s of rules to try to have Amazon parse the request. As a JS dev I can only guess this was designed by Java programmers where strict typing might suggest such a convoluted solution?
It took a while to figure out how to turn off all that stuff and just pass the request to my lambda JS based function.
I feel like API Gateway should be doing absolutely nothing except routing requests to the appropriate service. That should be the default and all that parsing should be a separate service. In other words it should be
API Gateway (no parsing) -> lambda
and if you want parsing then you'd
API Gateway -> Parsing service -> lambda
Or for that mater just make the parsing a library you can use if you want.
OTOH I'm not a network person so maybe there's valid reasons for API Gateway to have such a giant configuration
lambci has dockerized near perfect replicas of the lambda runtime environments, so as long as you aren’t relying on VPC endpoints tied to specific lambdas, or things like that, you should be able to, even without SAM.
But you can also: (1) deploy separate test copies of lambdas and fire test events at them the same way you would locally, which has full fidelity and no third-party deps. (2) throw test events at your lambda entry points locally and capture and validate what they return.
I do both regularly with python lambdas (I find that the latter is adequate for testing lambdas and their downstream interactions, but the former is useful for validating interactions upstream, particularly—for most of the lambdas I work with—with APIGateway and some auth stuff we have hooked in there.)
It gets reposted every year or so here on HN.
[1] http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.h... (2013)
I managed to get what I needed working but it was laborious compared to typical local server development.
I feel like most of the tools were written by devops folks because it’s quite easy to run these deployments in production, but the dev experience is lousy. The basics for developers is to be able to run the code locally and be able to run everything in a debugger. And that needs to include some sort of way to run services like dbs, queues etc. That’s just not possible as far as I can tell.
They say that cloud is somebody else’s server, well serverless is just somebody else’s monolith, and you don’t get a copy of it to develop on.
Not sure if it helps but here’s a demo IoT project I developed, has unit and integrations tests, it’s simple to understand, deployed using serverless:
https://github.com/mjgs/serverless-books-api
A few things I found useful:
netlify-cli - CLI tool for interacting with the Netlify API, great for deploying to Netlify when you aren’t building your site on their infrastructure.
netlify-lambda - CLI tool for building and running your Netlify functions locally.
serverless framework - A CLI and web tool to build and deploy serverless architectures, includes supports for Amazon, Azure, Google among others. Makes it easy to configure and provision all the components of your Cloud Native application.
These were taken from a blog post I wrote which some might find relevant:
Cloud Native web application development: what's it all about?
https://blog.markjgsmith.com/2021/01/12/cloud-native-web-app...
On the plus side of having a lot of the pieces in Azure, if you're lucky enough to have an input and output match (e.g. you want to persist SB Topic messages into a SQL API CosmosDB instance) it's a half dozen lines of code and you're done.
This reduces the importance of writing code and shifts focus into new development tools like AWS CDK. More and more of development work is about defining Step Functions logic and setting up various cloud events that hook all the pieces together.
It's not trivial to replicate ALL the hundreds of cloud services at home. So we accept that development is done in batches and deployment takes a while. It could be better if course, but it's not only about Lambda any more. It's about CloudFormation in general.
It sounds like your lambdas are calling other lambdas. If this is true, rethink your architecture.
If you have a thin, minimal code wrapper that translates your lambda invocation event into some sort of function invocation, it shouldn't be hard to test. Your external dependencies should be modeled as interfaces that can have fake implementations injected that either return pre-canned responses or implement things in memory.
Without more info about what you're doing, it makes it really hard to know what specifically is stopping you.
The problematic bit is the input event. You can't call real API Gtw. to trigger your Lambda code locally. You need to write/copy the request as JSON and provide it as input. It's more annoying with triggers like Kinesis Stream, but doable.
I personally don't like Localstack, it's safer to use real services.
The other challenge the framework solves is that payload wrappers are different if there is there is an sqs component in front of the lambda, or an sns in front of that. The execution code of the lamda should be completely agnostic to how the payload arrives.
https://docs.openfaas.com/ https://github.com/openfaas/faasd
We're actively developing this so any feedback of how we could make this better for your use case is welcome!
https://observablehq.com/@endpointservices/serverless-cells
Source code is loaded dynamically each invocation.
Still, some work todo, it's still annoying to program. Next hope to make them easy to use is pipe the runtime to Chrome devtools
The zip file packaging and debugging in the cloud has always frustrated me as well. But I like the docker packaging, still a long way to go but atleast I can somewhat mock some inputs and output triggers.
I have no idea how people managed the zip file uploads especially for python projects, it’s an insane waste of time uploading and debugging on the cloud.
For lambda based API's I am curious how it compares to cloudformation and terraform?
Then when I'm done, I push `lambda.Start(HandleFunc)`
Frequently, though, when I want to change some code I already have, I just run through the edit-build-push-invoke loop. It isn't optimal but with fast Internet it is not much slower than a Scala build and I can iterate with that.
Why not try to use this toolchain to local build/test your server less application
- Dashbird
- Thundra
- Lumigo
Cloud9 has direct Lambda integration, which helps too.
Processing kinesis streams, monitoring and munging S3 objects, auditing dynamo tables, etc. Web lambdas are not worth it imo, very complex and difficult to get right.
When building serverless applications, the most tested and iterated upon part of the application is our code which usually resides in a Lambda function. Testing Lambda functions breaks down to to angles. 1) invocation: testing services invoking a Lambda function, and 2) action: what is the Lambda function doing. This is the only part of the application that should be tested locally through local emulation. The rest of the application is best tested in the cloud.
IMHO the best way to test a Lambda function locally is with AWS SAM: https://aws.amazon.com/serverless/sam/
For testing invocation:
A Lambda function can only be invoked through the AWS Lambda service. SAM has three ways to emulate the Lambda service: 1) invoke - locally invoke the Lambda function one time [https://docs.aws.amazon.com/serverless-application-model/lat...] This functionality is helpful if you want to mock invoking a Lambda function from a service like S3, SNS, SQS, etc. Simply add an event. To create the proper event structure, SAM provides a command called generate event. [https://docs.aws.amazon.com/serverless-application-model/lat...] 2) start-lambda - start an emulator of the Lambda service that can be reached via SDK or AWS CLI [https://docs.aws.amazon.com/serverless-application-model/lat...] 3) start-api - start the Lambda service emulator with a basic API Gateway emulator wrapped around it. This creates a local endpoint for each Lambda function that uses an API GW event source [https://docs.aws.amazon.com/serverless-application-model/lat...]
For testing action:
Using one of the above commands will invoke a Lambda function. The Lambda function will run locally and provide logs as well as stepping through the code in an IDE like AWS Cloud9 or VS Code. The Lambda function can also call out to service like DynamoDB, SQS, SNS, etc that reside in the cloud. Once the Lambda function is working as expected locally, it's time to deploy to a development environment and run E2E tests.
One other tool I would suggest it SAM Logs. SAM Logs can output logs for a specific Lambda function from CloudWatch to your terminal. This is a great way to debug async Lambda functions in the cloud.
I encourage you to visit https://serverlessland.com where we are constantly adding content to help developers with Serverless on AWS. I also have a series of SAM videos at https://s12d.com/sws. Additionally, we host Serverless Office Hours every Tuesday: https://twitch.tv/aws or https://youtube.com/serverlessland. During this time we answer any and all serverless related questions.
Having said all this. Our team is continuing to work towards making the development experience better. Threads like this are critical to our understanding of developer needs and we read them and take them to heart. If you would like to have a longer conversation please reach out to me at ericdj@amazon.com or @edjgeek on Twitter.
I try to use the AWS 'resource API' rather than 'service API', since it's usually easier to understand. The latter can do anything, but deals with fiddly 'Request' and 'Response' values; the former isn't as expansive, but provides high-level things like 'Tables', 'Buckets', etc.
I wrap all calls to AWS in a failure mechanism, and check for nulls immediately. I usually use Scala's `Try[T]` type, which is essentially `Either[Exception, T]`. Note that there are some cases where null is expected, like an empty DynamoDB.get result. Those should be turned into `Try[Option[T]]` values immediately.
I'll aggressively simplify the interface that an application depends on. For example, a Lambda might be completely independent of DynamoDB except for a single function like:
put: Row => Try[Unit]
I'll usually wrap these simple type signatures in an interface, with an AWS-backed implementation and a stub implementation for testing (usually little more than a HashMap). These stubs can usually be put in a shared library and re-used across projects.
My current approach to dependency injection is to dynamically bind the overridable part (e.g. using a scala.util.DynamicVariable). This can be hidden behind a nicer API. The "real" AWS-backed version is bound by default; usually wrapped in a `Try` or `Future`, to prevent failures from affecting anything else.
All business/application logic is written against this nice, simple API. Tests can re-bind whatever they need, e.g. swapping out the UserTable with a HashMap stub.
I tend to use property-based tests, like ScalaCheck, since they're good at finding edge-cases, and don't require us to invent test data by hand.
For each project I'll usually write a "healthcheck" lambda. This returns various information about the system, e.g. counting database rows, last-accessed times, etc. as well as performing integration tests (e.g. test queries) to check that the AWS-backed implementations work, that we can connect to all the needed systems, etc.