Loading...

NVRam for Fun and Profit Shaun Thomas COTILLION.9.15.11

123 views

Loading...

Loading...

Transcript

The interactive transcript could not be loaded.

Loading...

Loading...

Rating is available when the video has been rented.
This feature is not available right now. Please try again later.
Published on Mar 7, 2012

This talk is part case study, part solution, for exceedingly transactional databases that depend on high throughput and low latency. It is how OptionsHouse operates a 10,000+ TPS database without cache-warming or any other convoluted magic background scripts. Learn how to break the TPS barrier imposed by traditional RAID deployments.
Abstract

One common problem with PostgreSQL, as with any database, is IO contention due to random access patterns. A lot of companies seek to solve this problem by adding more and more memory so OS cache hits supplant disk access. But there's a hidden danger here: cache poisoning. Any kind of cache invalidation, from table dumps, system backup, or regular vacuums, can bump critical blocks out of memory. When this happens to an OLTP database, performance can drop to a mere fraction of the necessary level to keep an application online.

For most of 2010, OptionsHouse's volume of data accrual and interaction completely saturated the existing RAID solution. Tripling system memory and cache preloading helped alleviate this temporarily, but proved a fragile and crude solution; any maintenance or a rogue select statement quickly reduced TPS from 10,000 to less than 2,000. Eventually NVRAM solutions were tested for their touted random read and write IOPS, in an attempt to halt the frequent front-end timeouts caused by lagging IO.

These tests suggested that a single PCI card could outperform the existing RAID by more than an order of magnitude. Subsequent application log replays proved production usage patterns fit these results, even when constrained to a single LUN and throttled by DRBD replication.

Loading...

When autoplay is enabled, a suggested video will automatically play next.

Up next


to add this to Watch Later

Add to

Loading playlists...