Asynchronous Task Queues
Background
What
- Work performed in another process/thread
- Work performed on another machine or a pool of machines
Why
- Increase User-perceived performance
- Avoid IO limitations (slow, unreliable, etc)
- Scale horizontally
- Avoid GIL in some languages (Ruby, Python)
When
- Work considered optional at the time of execution
- Web development: action(s) not required to build HTTP response
Basic Concept
- Client pushes message onto queue
- Worker shifts message off queue
- Worker executes code based on message
Architecture
Message Serialization
- Structured payload format
- Cross-platform type handling
- Speed / Payload size
Examples
- JSON
- msgpack
- YAML
Abstractions
- Generic interface to many job processors
- Allow libraries to perform async work
Examples
- Active Job
- ???
Strategies
Pub/Sub
- Most message brokers (& PG with LISTEN/NOTIFY)
- Less chance of race conditions - one message per worker
- CPU efficient - block until recv
Polling
- Normal with Database message bus
- Race condition prone
Job processors
- Batteries included
- Consistent interface
- Job retry / worker durability
- Some have concurrency built in
| Language | Message Broker | |
|---|---|---|
| Celery | Python | Redis, Beanstalk, MongoDB, CouchDB, Database |
| Sidekiq | Ruby | Redis |
| Resque | Ruby | Redis |
| Delayed Job | Ruby | Database |
Message Brokers
- Accept messages from clients
- Determine possible worker via queue names
- Notify worker of job
Examples
- Beanstalk
- RabbitMQ
- Redis
- Database
- IronMQ
- Amazon SQS
- many, many more
Best Practices
- Use small stateless messages
- Keep data out of queue
- Queue should have actions
- Use a simple serialized type (JSON?)
- Idempotent and transactional actions
Demo
Thank You
- Github
- github.com/adam12
- @adamrdaniels
- Sourcecode
- github.com/adam12/asynchronous-task-queues-devtricks-2014
/