Realtime Distributed Message Processing at Scale with NSQ





The interactive transcript could not be loaded.


Rating is available when the video has been rented.
This feature is not available right now. Please try again later.
Published on Jun 18, 2013

Don’t miss the next DataEngConf in Barcelona: https://dataeng.co/2O0ZUq7

Video Intro Animation provided by http://animoto.com/

Visit http://www.g33ktalk.com for more videos and tech talks. NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. It is the backbone of bitly's infrastructure, powering realtime data analysis and feeding offline batch processes.

Bios: Matt Reiferson is a software engineer, lunch instructor, and fan of tv's Bob Ross. Matt is an Engineer at bitly, one of the main architects behind NSQ. He is most known as the creator and host of The Joy of Coding in DOS, a television program that ran for twelve years on PBS stations in the United States.

Jehiah is a co-author of NSQ. When not painstakingly shortening urls by hand, Jehiah can be found riding trains, learning Amharic, or building iPhone apps to help others do both of those things more efficiently.

Some background on NSQ

NSQ -- realtime distributed message processing at scale. Let's examine what that means. NSQ is used in the backend of software architectures to process large quantities of messages. The scale NSQ is designed to handle is in the order of billions of messages per day or even per hour. It is designed to do that without a single point of failure which means that even at incredible scale and load, it is still extremely reliable. NSQ does not depend on data structure of a database. It can process data in JSON forma or a number of other protocols.

And right from the Bitle website, here is what NSQ is designed to do:

provide easy topology solutions that enable high-availability and eliminate SPOFs
address the need for stronger message delivery guarantees
bound the memory footprint of a single process (by persisting some messages to disk)
greatly simplify configuration requirements for producers and consumers
provide a straightforward upgrade path to improve efficiency.

Here are the sliders from the presentation:

And here is the full blog post: http://g33ktalk.com/bitly-nsq/


When autoplay is enabled, a suggested video will automatically play next.

Up next

to add this to Watch Later

Add to

Loading playlists...