310
Percona Server 5.7 並行doublewrite 特性
In this blog post, we’ll discuss the ins and outs of Percona Server 5.7 parallel doublewrite.
在這篇文章中，我們將由裏及外討論Percona Server 5.7的並行doublewrite。
After implementing parallel LRU flushing as described in the previous post, we went back to benchmarking. At first, we tested with the doublewrite buffer turned off. We wanted to isolate the effect of the parallel LRU flusher, and the results validated the design. Then we turned the doublewrite buffer back on and saw very little, if any, gain from the parallel LRU flusher. What happened? Let’s take a look at the data:
在上篇文章（Percona Server 5.7: multi-threaded LRU flushing，詳見文末：延伸閱讀）中 ，我們描述了多線程LRU刷新線程的實現，現在讓我們回到基準測試。首先，在doublewrite buffer關閉狀態下進行測試。我們想要隔離並行LRU刷新帶來的影響，結果驗證了設想。然後，重新開啟doublewrite，發現從並行LRU刷新獲得的好處很小。到底發生了什麼？我們先來看看數據：
We see that the doublewrite buffer mutex is gone as expected and that the top waiters are the rseg mutexes and the index lock (shouldn’t this be fixed in 5.7?). Then we checked PMP：
如上圖，我們看到doublewrite buffer互斥量如預期一樣消失了，最高的等待是rseg 互斥量和index鎖（這不應該在5.7修複了麼？）。接著，我們檢查下PMP：
Again we see that PFS is not telling the whole story, this time due to a missing annotation in XtraDB. Whereas the PFS results might lead us to leave the flushing analysis and focus on the rseg/undo/purge or check the index lock, PMP clearly shows that a lack of free pages is the biggest source of waits. Turning on the doublewrite buffer makes LRU flushing inadequate again. This data, however, doesn’t tell us why that is.
我們再次看到，PFS並沒有顯示所有的內容，這裏是因為XtraDB的缺陷（詳見文末延伸閱讀）。而PFS的結果讓我們忽略刷新方麵的分析，轉而聚焦rseg/undo/purge或者索引鎖的檢查上。PMP清晰地展現缺少空閑也是最大等待的源頭。打開doublewriter buffer又會導致LRU刷新不足。然而，這些數據並沒有告訴我們為什麼會這樣。
To see how enabling the doublewrite buffer makes LRU flushing perform worse, we collect PFS and PMP data only for the server flusher (cleaner coordinator, cleaner worker, and LRU flusher) threads and I/O completion threads:
為了了解為何開啟了doublewrite buffer 會使LRU刷新變得糟糕，我們收集了PFS和PMP數據，這些數據隻包含刷新相關（cleaner coordinator，cleaner worker，以及LRU flusher）線程和I/O相關線程：
If we zoom in from the whole server to the flushers only, the doublewrite mutex is back. Since we removed its contention for the single page flushes, it must be the batch doublewrite buffer usage by the flusher threads that causes it to reappear. The doublewrite buffer has a single area for 120 pages that is shared and filled by flusher threads. The page add to the batch action is protected by the doublewrite mutex, serialising the adds, and results in the following picture:
如果我們從整個服務放大到刷新線程，就又能看到douoblewrite mutex了。由於我們移除了單頁刷新之間的爭用，所以它會在刷新線程批量使用doublewrite buffer時重新出現。doublewrite buffer有一個有120個page的單獨區域，刷新線程負責填充並共享使用。將頁添加到批處理操作由doublewrite mutex保護，持續添加之後的結果如下圖：
By now we should be wary of reviewing PFS data without checking its results against PMP. Here it is:
現在我們應該更謹慎地評估PFS數據，並與PMP進行對比。PMP結果如下：
As with the single-page flush doublewrite contention and the wait to get a free page in the previous posts, here we have an unannotated-for-Performance Schema doublewrite OS event wait (same bug 80979):
與之前文章中提到的單頁刷新doublewrite爭用，等待一個空閑的頁的情景一樣，這裏我們有一個在Performance Schema中未被注解的doublewrite OS 事件。
This is as bad as it looks (the comment is outdated). A running doublewrite flush blocks any doublewrite page add attempts from all the other flusher threads for the duration of the flush (up to 120 data pages written twice to storage)：
這看起來很糟糕（裏麵的注釋可以不用關注，已經過時）。活躍的doublewrite刷新時會阻塞所有其他flush線程任何的doublewrite page添加（多達120個頁寫入兩次存儲）：
The issue also occurs with MySQL 5.7 multi-threaded flusher but becomes more acute with the PS 5.7 multi-threaded LRU flusher. There is no inherent reason why all the parallel flusher threads must share the single doublewrite buffer. Each thread can have its own private buffer, and doing so allows us to add to the buffers and flush them independently. This means a lot of synchronisation simply disappears. Adding pages to parallel buffers is fully asynchronous:
使用MySQL 5.7多線程flush也會出現此問題，但Percona Server 5.7的多線程LRU 刷新尤為突出。但並發flush線程並非必須共享單個doublewrite buffer。每個線程都可以有自己的私有buffer，這樣可以允許添加到buffer並單獨刷新它們。這意味著大量的同步會消失。將頁麵添加到並行buffer完全是異步的。
And so is flushing them:
變成了下麵的刷新模式：
This behavior is what we shipped in the 5.7.11-4 release, and the performance results were shown in a previous post. To see how the private doublewrite buffer affects flusher threads, let’s look at isolated data for those threads again.
這個特性是我們在5.7.11-4版本添加的，其性能提升效果在之前的文章(《Percona Server 5.7 performance improvements》）中已經展示。想知道私有doublewrite buffer對flush線程的影響，讓我們再看下這些線程的隔離數據：
Performance Schema:
It shows the redo log mutex as the current top contention source from the PFS point of view, which is not caused directly by flushing.
從PFS的角度看，redo log互斥量是當前使用量最多的爭用來源，這不是直接由flush引起的。
PMP data looks better too:
PMP的數據看起來好點：
The buf_dblwr_flush_buffered_writes now waits for its own thread I/O to complete and doesn’t block other threads from proceeding. The other top mutex waits belong to the LRU list mutex, which is again not caused directly by flushing.
buf_dblwr_flush_buffered_writes 現在等待自己的線程I/O完成，並不阻塞其他線程運行。其他較高的互斥量等待屬於LRU list mutex，這也不是由flush引起的。
This concludes the description of the current flushing implementation in Percona Server. To sum up, in these post series we took you through the road to the current XtraDB 5.7 flushing implementation:
- Under high concurrency I/O-bound workloads, the server has a high demand for free buffer pages. This demand can be satisfied by either LRU batch flushing, either single page flushing.
- Single page flushes cause a lot of doublewrite buffer contention and are bad even without the doublewrite.
- Same as in XtraDB 5.6, we removed the single page flushing altogether.
- Existing cleaner LRU flushing could not satisfy free page demand.
- Multi-threaded LRU flushing design addresses this issue – if the doublewrite buffer is disabled.
- If the doublewrite buffer is enabled, MT LRU flushing contends on it, negating its improvements.
- Parallel doublewrite buffers address this bottleneck.
以下是對當前Percona Server的flush實現描述。總結一下，在這些的係列文章中，我們重現了XtraDB 5.7刷新實現的風雨曆程：
- 在I/O密集的工作負載下，server對空閑buffer頁麵的需求量很大，這要求通過批量的LRU刷新，或者單個頁麵刷新來滿足需要。
- 單個頁麵flush會導致大量的doublewrite buffer爭用，即使沒開啟doublewrite也是糟糕的。
- 與XtraDB 5.6 相同，我們一並移除了單頁flush。
- 現有的LRU刷新機製不能滿足空閑頁的需求。
- 多線程LRU刷新解決了這個問題——如果doublewrite buffer關閉掉。
- 如果開啟了doublewrite,則多線程的LRU flush會爭用它,也會使得性能下降。
- 並行的doublewrite buffer解決了這個問題。
原文發布時間為：2017-10-16
原文作者：Laurynas Biveinis and Alexey Stroganov
翻譯團隊：知數堂藏經閣項目 - 天一閣
本文來自雲棲社區合作夥伴“老葉茶館”，了解相關信息可以關注“老葉茶館”微信公眾號
最後更新：2017-10-17 16:33:47