Visit http://www.g33ktalk.com for more videos and tech talks. NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It is the backbone of bitly's infrastructure, powering realtime data analysis and feeding offline batch processes.
Bios: Matt Reiferson is a software engineer, lunch instructor, and fan of tv's Bob Ross. Matt is an Engineer at bitly, one of the main architects behind NSQ. He is most known as the creator and host of The Joy of Coding in DOS, a television program that ran for twelve years on PBS stations in the United States.
Jehiah is a co-author of NSQ. When not painstakingly shortening urls by hand, Jehiah can be found riding trains, learning Amharic, or building iPhone apps to help others do both of those things more efficiently.
Some background on NSQ
NSQ -- realtime distributed message processing at scale. Let's examine what that means. NSQ is used in the backend of software architectures to process large quantities of messages. The scale NSQ is designed to handle is in the order of billions of messages per day or even per hour. It is designed to do that without a single point of failure which means that even at incredible scale and load, it is still extremely reliable. NSQ does not depend on data structure of a database. It can process data in JSON forma or a number of other protocols.
And right from the Bitle website, here is what NSQ is designed to do:
provide easy topology solutions that enable high-availability and eliminate SPOFs address the need for stronger message delivery guarantees bound the memory footprint of a single process (by persisting some messages to disk) greatly simplify configuration requirements for producers and consumers provide a straightforward upgrade path to improve efficiency.