I've been throwing a lot of my own personal resources into building some things on top of both S3 and lambda and have found a few tools that help with it quite a bit. For one - the lambduh project from Russ Matney has been a great resource for abstracting out some of the more common s3->lambda workflow: https://github.com/lambduh/lambduh. On a different note is T.J. Holowaychuk's Apex project: https://medium.com/@tjholowaychuk/introducing-apex-800824ffa...
How many HTTP requests can Lambda do concurrently? Is my best approach to fire all these requests inside one worker, or should/could I have it spin up subsequent lambadas whose only function is to run the HTTP request then close? I'm imagining that would be a lot more expensive.
Should you choose the fanout route, Tim Wagner from AWS told me that it's pretty fast: https://twitter.com/timallenwagner/status/658025794900365312
So, alternatively, you could do something with DynamoDB event sources, where you have some sort of pub/sub table that your lambda functions listen on (basically a list of all the http requests that have to happen) - thus keeping a minimal 1 lambda dispatch per http request. The catch is you would need another system to manage that table (technically that system can be lambda itself).
Two important things, 1) I haven't used the dynamodb/lambda integration myself so be skeptical of my suggestion and 2) what I can say from our usage of the s3/lambda integration is that concurrency is not a problem with thousands of lambda dispatches/second being surprisingly quick to spin up.
but found one issue, that many here might not be aware of, S3 bucket and Lambda function should be in same aws region.
Unfortunately, my s3 bucket is in southeast-ap, and aws lambda is not available in this region. couldn't go live today. will have to copy bucket to another region to use it.
hope this helps. thanks.
require 'java'
java_import 'com.amazonaws.services.lambda.runtime.Context'
java_import 'java.util.Map'
class Main
java_signature 'static String handler(Map<String,String> args, Context context)'
def self.handler(args, context)
puts "hello world"
end
end
# jrubyc --java main.rb