index.json

[{"content":"This is also posted to the mailing list.\nHello community,\nI shared DAMON quarterly news [1, 2, 3, 4] in 2024. I was further planning to write that for 2025 Q1. But I dropped the ball and never finished it. I\u0026rsquo;m starting a new series of DAMON news, that aims to cover each -rc1 release.\nThis news letter covers DAMON features that landed on the mainline, and misc events that were interesting to me that happened in v6.19-rc1..v7.0-rc1 time period (2025-12-14..2026-02-22).\nTL; DR Interesting DAMON development ideas including PMU-based extension, automated THP application, and new DAMOS quota auto-tune metrics for CXL-based tiered memory systems have been proposed.\nTwo DAMON topics for LSF/MM/BPF are proposed.\nIn the time period, 12 grateful contributors landed 75 commits on the linux mainline.\nNotable Events In the time period, many events happened. Among those, I think below events are worthy to be noted here.\nRelease Seven DAMON hotfixes have landed on 6.19.\n$ git log v6.19-rc1..v6.19 --oneline -- mm/damon include/linux/damon.h \\ Documentation/mm/damon/ Documentation/admin-guide/mm/damon/ \\ include/trace/events/damon.h samples/damon/ \\ tools/testing/selftests/damon | wc -l 7 Sixty six DAMON changes have landed on 7.0-rc1 during the merge window.\n$ git log v6.19..v7.0-rc1 --oneline -- mm/damon include/linux/damon.h \\ Documentation/mm/damon/ Documentation/admin-guide/mm/damon/ \\ include/trace/events/damon.h samples/damon/ \\ tools/testing/selftests/damon | wc -l 66 Read below \u0026lsquo;Changes Landed On \u0026hellip;\u0026rsquo; sections for more details of the changes including user-visible ones (mostly new features).\nNew and Ongoing Developments Enze Li proposed [5] to make DAMON supports loadable modules. Because of the lack of the real use case, it is on hold for now.\nAsier proposed [6] a new DAMON module that finds hot processes and applies THP to hot pages of the processes. We are aligned on the high level approach, though details might need some changes. Asier is further working on tests.\nAkinobu Mita proposed [7] PMU-based access check. It could allow sub-page level access monitoring and more interesting expansions of DAMON, such as per-CPUs/processes/reads/writes monitoring. It may be restrictive for virtual machines, though. Akinobu is looking at rebasing it on my page faults-based similar extension RFC code.\nRavi Jonnalagadda proposed [8] new DAMOS quota goal metrics for weights-based pages promotion and demotion on CXL-based tiered memory systems. After a few revisioning, I think we are now quite aligned on the direction of the change. Ravi continues working on it, with tests.\nLSF/MM/BPF DAMON is picked [9] as one of Linux kernel machine learning library demonstration use cases. DAMON community promised any collaborations for that on demand.\nTwo DAMON-related LSF/MM/BPF topics are suggested. The first one [10] aims to discuss possible DAMON usage for THP, on production level. It will share past DAMON-based THP improvement works and why it was inappropriate on product environments. Further, it will brainstorm better approaches. Asier\u0026rsquo;s work may also be discussed.\nThe second one [11] aims to make alignment on required DAMON-external changes for page faults based per-CPUs/threads/reads/writes DAMON monitoring extension. The very initial extension attempt was started from nearly the very beginning of DAMON, and I continued it from last summer. Hopefully it will make more progress or happily fail early.\nMisc DAMON user-space tool (damo) started supporting trace-cmd [12], making use of it on Android much easier. It also started supporting command line auto-completion [13].\nDAMON development yearly retrospects for 2024 [14] and 2025 [15] are also finally posted.\nStatistics In the time period, 12 people made 75 grateful DAMON commits on the Linux mainline. The script is availabe at my GitHub toolbox repo [16].\n$ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --linux_subsystems DAMON --since 2025-12-14 --until 2026-02-22 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 54 commits 2. Shu Anzai \u0026lt;shu17az@gmail.com\u0026gt;: 5 commits 3. Enze Li \u0026lt;lienze@kylinos.cn\u0026gt;: 3 commits 4. Shakeel Butt \u0026lt;shakeel.butt@linux.dev\u0026gt;: 3 commits 5. Kees Cook \u0026lt;kees@kernel.org\u0026gt;: 2 commits 6. Linus Torvalds \u0026lt;torvalds@linux-foundation.org\u0026gt;: 2 commits 7. Li RongQing \u0026lt;lirongqing@baidu.com\u0026gt;: 1 commits 8. Aaron Yang \u0026lt;yangqixiao@inspur.com\u0026gt;: 1 commits 9. Kevin Lourenco \u0026lt;klourencodev@gmail.com\u0026gt;: 1 commits 10. JaeJoon Jung \u0026lt;rgbi3307@gmail.com\u0026gt;: 1 commits 11. Swaraj Gaikwad \u0026lt;swarajgaikwad1925@gmail.com\u0026gt;: 1 commits 12. Akinobu Mita \u0026lt;akinobu.mita@gmail.com\u0026gt;: 1 commits # 12 authors, 75 commits in total DAMON user-space tool also got grateful changes.\n$ ./lazybox/version_control/authors.py ./damo --skip_merge_commits \\ --since 2025-12-14 --until 2026-02-22 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 335 commits 2. Andrew Paniakin \u0026lt;apanyaki@amazon.com\u0026gt;: 2 commits # 2 authors, 337 commits in total According to \u0026lsquo;hkml\u0026rsquo; [20], DAMON mailing list received 440 mails of 111 threads. 77 threads were newly initiated. Among the mails, 223 mails were for patches. It included 43 series.\n$ hkml list damon --alias tmp --since v6.19-rc1 --until v7.0-rc1 --collapse --stdout --stat_only # (cached output) # 440 mails, 111 threads, 72 new threads # 223 patches, 43 series # oldest: 2025-12-11 04:56:14-08:00 # newest: 2026-02-22 13:01:01-08:00 Huge appreciation to all the great contributors!\nChanges landed on v6.19 All DAMON changes that landed on v6.19-rc1 via MM pull requests [17, 18] are successfully landed on v6.19.\nUser-visible Changes Among the changes, below two new features are notable.\ncommit adf7d6cdd716e: Patch series \u0026quot;mm/damon: support pin-point targets removal\u0026quot;. commit d3946c5f4c1c5: Patch series \u0026quot;mm/damon: allow DAMOS auto-tuned for per-memcg per-node memory usage.\u0026quot; The first feature allows more efficient and flexible use of DAMON and DAMOS for dynamic list of virtual address spaces. It was motivated by Bijan\u0026rsquo;s dynamic access-aware memory interleaving on CXL-based memory tiering work.\nThe second feature is for general per-memcg per-node memory utilization based automatic access-aware system operations. The motivation was per-cgroup fairness on memory tiering use case.\nUser-invisible Changes Below non-feature changes that made for more reliable and complete DAMON tests are also notable. The kunit tests memory bugs fix is now on all relevant stable kernels.\ncommit b5ab490d85b77: Patch series \u0026quot;mm/damon/tests: fix memory bugs in kunit tests\u0026quot;. commit e859a224fad65: Patch series \u0026quot;mm/damon: fixes for address alignment issues in DAMON_LRU_SORT and DAMON_RECLAIM\u0026quot;. commit 37104286f9390: Patch series \u0026quot;mm/damon/tests: add more tests for online parameters commit\u0026quot;. Changes landed on v7.0-rc1 During the 7.0-rc1 merge window, 66 DAMON patches have landed via the MM pull request [19].\nUser-visible Changes In the merge window, below user-visible new features are added.\ncommit 4835e2871321f: Patch series \u0026quot;mm/damon: advance DAMOS-based LRU sorting\u0026quot;. commit 4a6ceb7c9744c: Patch series \u0026quot;mm/damon: introduce {,max_}nr_snapshots and tracepoint for damos stats\u0026quot;. The first series advances DAMON_LRU_SORT module, which protects hot LRU pages, to be easier to control. Support of active:inactive list size based automation and DAMON monitoring intervals automation are especially notable.\nThe second series allows more deterministic control and internal behavior observabilities for DAMOS. The development was motivated by Ravi\u0026rsquo;s dynamic page interleaving latency/capacity improvement-aimed CXL memory tiering approach.\nUser-invisible Changes Also important changes for DAMON tests improvement, API/code code refactoring, and documentation cleanup were landed, including below series.\ncommit 94a62284ede02: Patch series \u0026quot;selftests/damon: improve leak detection and wss estimation reliability.\u0026quot; commit 6c59085fc0942: Patch series \u0026quot;mm/damon/tests/core-kunit: extend existing test scenarios\u0026quot;, commit 4262c53236977: Patch series \u0026quot;mm/damon: hide kdamond and kdamond_lock from API callers\u0026quot;. commit 50962b16c0d63: Patch series \u0026quot;mm/damon: cleanup kdamond, damon_call(), damos filter and DAMON_MIN_REGION.\u0026quot; commit 32d11b3208971: Patch series \u0026quot;Docs/mm/damon: update intro, modules, maintainer profile, and misc.\u0026quot; References [1] 2024 Q1 news: https://lore.kernel.org//20240402191224.92305-1-sj@kernel.org/\n[2] 2024 Q2 news: https://lore.kernel.org/20241001002449.515043-1-sj@kernel.org/\n[3] 2024 Q3 news: https://lore.kernel.org/20241001191425.588219-1-sj@kernel.org/\n[4] 2024 Q4 news: https://lore.kernel.org/20250102211811.48322-1-sj@kernel.org/\n[5] loadable module support: https://lore.kernel.org/20251215142057.588500-1-lienze@kylinos.cn\n[6] THP-ing hot processes: https://lore.kernel.org/20260202145650.1795854-1-gutierrez.asier@huawei-partners.com\n[7] perf event based access check: https://lore.kernel.org/20260123021014.26915-1-akinobu.mita@gmail.com\n[8] node_target_[in]eligible_mem_bp: https://lore.kernel.org/20260123045733.6954-1-ravis.opensrc@gmail.com\n[9] ML library: https://lore.kernel.org/all/c24a209d5a4af0c4cc08f30098998ce16c668b58.camel@ibm.com/\n[10] DAMON for THP, LSFMMBPF: https://lore.kernel.org/all/20260211022845.68865-1-sj@kernel.org/\n[11] DAMON for faults, LSFMMBPF: https://lore.kernel.org/all/20260218054320.4570-1-sj@kernel.org/\n[12] damo trace-cmd support: https://damonitor.github.io/posts/damo_support_trace_cmd/\n[13] damo cli auto-completion: https://github.com/damonitor/damo/blob/next/USAGE.md#command-line-auto-completion\n[14] DAMON yearly restrospects, 2024: https://lore.kernel.org/all/20260216210625.68098-1-sj@kernel.org/\n[15] DAMON yearly retrospects, 2025: https://lore.kernel.org/all/20260222210102.153347-1-sj@kernel.org/\n[16] https://github.com/sjp38/lazybox/blob/master/version_control/authors.py\n[17] 6.19-rc1 mm pr 1/2: https://lore.kernel.org/20251203212918.82f1c9d3947940aeae263878@linux-foundation.org/\n[18] 6.19-rc1 mm pr 2/2: https://lore.kernel.org/20251211131127.defed99e5b82e49af605108a@linux-foundation.org/\n[19] 7.0-rc1 mm pr: https://lore.kernel.org/20260211192351.6684a77b8c70cc032a3e7a27@linux-foundation.org/\n[20] https://docs.kernel.org/process/email-clients.html#hackermail-tui\nThanks, SJ\n","permalink":"https://damonitor.github.io/posts/release_news_7.0_rc1/","summary":"This is also posted to the mailing list.\nHello community,\nI shared DAMON quarterly news [1, 2, 3, 4] in 2024. I was further planning to write that for 2025 Q1. But I dropped the ball and never finished it. I\u0026rsquo;m starting a new series of DAMON news, that aims to cover each -rc1 release.\nThis news letter covers DAMON features that landed on the mainline, and misc events that were interesting to me that happened in v6.","title":"DAMON Release News (v6.19-rc1..v7.0-rc1)"},{"content":"This is also posted to DAMON mailing list.\nIt is a bit late, but let me share my retrospect of DAMON development for 2025, before my memory goes away. The yearly retrospects for 2022 [1], 2023 [2] and 2024 [3] are also available.\nSummary 2025 was the busiest year for DAMON development of its history. 33 people made 390 commits for DAMON in 2025. Those numbers are 65 % and 157 % increase from those of 2024. Before 2025, 2022 was the busiest year that 32 people made 210 commits for DAMON.\nContributions from Meta except myself (observabilities for hugepages and LRU), Micron (access-aware runtime memory interleaving for CXL memory), Huawei (arm32 LPAE support), SK hynix (memory tiering related fixes) and individuals (important fixes and refactoring) were impressive and notable.\nMajor motivations that led DAMON development in 2025 were self-tuning, system/fleet observability and memory tiering.\nEvents There were many events in 2025. Introducing only a few notable ones. A more exhaustive list of events is also available at DAMON project site [4].\nAdoptions In April, OpenSuse kernels build-enabled [5] DAMON.\nIn June, DAMON was build-enabled on mainline by default [6], and reverted [7] by Linus Torvalds.\nIn October, DAMON module for system/fleet-wide access monitoring (DAMON_STAT) has build-enabled [8] on Debian kernels.\nDAMON Feature Releases Many DAMON features have developed and landed on the mainline in 2025. Listing only a few of those that stands out to me. A more exhaustive list of features is available on DAMON user-space tool\u0026rsquo;s DAMON features list [9].\nIn February, Linux 6.14-rc1 was released, with page level monitoring [10] feature.\nIn April, Linux 6.15-rc1 was released. Usama and Naht from Meta contributed huge pages [11] and LRU pages [12] monitoring/operation extensions. The release also introduced the monitoring intervals goal auto-tuning [13], which made DAMON just work.\nIn June, Linux 6.16-rc1 was released, with DAMON\u0026rsquo;s memory tiering automation support [14]. The feature is not only for memory tiering but general per-NUMA utilization based access-aware system operations automation, though.\nIn August, Linux 6.17-rc1 was released. In this release, Bijan and Ravi from Micron contributed access-aware runtime memory interleaving features [15] for their CXL memory tiering solution. A module for fleet-wide access monitoring (DAMON_STAT) [16] is also introduced.\nIn October, Linux 6.18-rc1 was released. In this release, Quanmin from Huawei contributed [17] arm32 Large Physical Address Extension support. Yueyang Pan from Meta contributed [18] huge pages observability on virtual address space.\nIn December, Linux 6.19-rc1 was released. In this release, per-node per-memcg memory utilization based DAMOS automation [19] is introduced. Motivation of the feature was per-cgroup memory tiering.\nPresentations 2025 was an interesting year in terms of presentation, since I found two presentations from someone other than myself that introduced DAMON use cases.\nIn October, Asier from Huawei presented a DAMON use case for THP [20] at OSSummit EU in August.\nIn November, Ravi from Micron presented a DAMON use case for CXL memory tiering aiming both latency and bandwidth improvements [21] at Linux Memory Hotness and Promotion meeting.\nI also led some sessions at conferences in 2025.\nIn February, I presented DAMON in whole, at FOSDEM [22].\nIn March, I led two sessions discussing DAMON development status and future at LSF/MM/BPF [23].\nIn June, I introduced DAMON\u0026rsquo;s new intervals autotuning feature [24] at OSSummit NA.\nIn September, I introduced DAMON\u0026rsquo;s monitoring mechanism and usage [25] at Kernel Recipes.\nIn December, I led two DAMON discussion sessions [26,27] and one presentation [28] at LPC special-purpose memory micro-conference, system monitoring micro-conference, and refereed track, respectively.\nStatistics Like previous yearly retrospects, I ran my script [29] to see DAMON development statistics as below. In 2025, 33 people made 390 commits for DAMON. Huge appreciation to all grateful contributors.\n$ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --linux_subsystems DAMON --year 2025 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 310 commits 2. Quanmin Yan \u0026lt;yanquanmin1@huawei.com\u0026gt;: 10 commits 3. Bijan Tabatabai \u0026lt;bijantabatab@micron.com\u0026gt;: 10 commits 4. Enze Li \u0026lt;lienze@kylinos.cn\u0026gt;: 8 commits 5. Sang-Heon Jeon \u0026lt;ekffu200098@gmail.com\u0026gt;: 8 commits 6. Honggyu Kim \u0026lt;honggyu.kim@sk.com\u0026gt;: 6 commits 7. Usama Arif \u0026lt;usamaarif642@gmail.com\u0026gt;: 6 commits 8. Akinobu Mita \u0026lt;akinobu.mita@gmail.com\u0026gt;: 3 commits 9. Yueyang Pan \u0026lt;pyyjason@gmail.com\u0026gt;: 2 commits 10. Yunjeong Mun \u0026lt;yunjeong.mun@sk.com\u0026gt;: 2 commits 11. Nhat Pham \u0026lt;nphamcs@gmail.com\u0026gt;: 2 commits 12. David Hildenbrand \u0026lt;david@redhat.com\u0026gt;: 2 commits 13. Arnd Bergmann \u0026lt;arnd@arndb.de\u0026gt;: 1 commits 14. Dan Carpenter \u0026lt;dan.carpenter@linaro.org\u0026gt;: 1 commits 15. Lorenzo Stoakes \u0026lt;lorenzo.stoakes@oracle.com\u0026gt;: 1 commits 16. Balbir Singh \u0026lt;balbirs@nvidia.com\u0026gt;: 1 commits 17. Swaraj Gaikwad \u0026lt;swarajgaikwad1925@gmail.com\u0026gt;: 1 commits 18. Lokesh Gidra \u0026lt;lokeshgidra@google.com\u0026gt;: 1 commits 19. jianyun.gao \u0026lt;jianyungao89@gmail.com\u0026gt;: 1 commits 20. Alexandre Ghiti \u0026lt;alexghiti@rivosinc.com\u0026gt;: 1 commits 21. Qianfeng Rong \u0026lt;rongqianfeng@vivo.com\u0026gt;: 1 commits 22. Stanislav Fort \u0026lt;stanislav.fort@aisle.com\u0026gt;: 1 commits 23. Bjorn Helgaas \u0026lt;bhelgaas@google.com\u0026gt;: 1 commits 24. Nathan Gao \u0026lt;zcgao@amazon.com\u0026gt;: 1 commits 25. Linus Torvalds \u0026lt;torvalds@linux-foundation.org\u0026gt;: 1 commits 26. Thushara.M.S \u0026lt;thusharms@gmail.com\u0026gt;: 1 commits 27. Su Hui \u0026lt;suhui@nfschina.com\u0026gt;: 1 commits 28. Taotao Chen \u0026lt;chentaotao@didiglobal.com\u0026gt;: 1 commits 29. Seongjun Kim \u0026lt;bus710@gmail.com\u0026gt;: 1 commits 30. Marcelo Moreira \u0026lt;marcelomoreira1905@gmail.com\u0026gt;: 1 commits # 33 authors, 390 commits in total It is a dramatic increase of the amount from that of 2024. In 2024, 20 people made 157 commits. So 65 % increase of developers, and 148 % increase of commits.\n2025 was not just busier than 2024. It was the busiest year in DAMON\u0026rsquo;s history. The stats for the whole history of DAMON are as below.\n$ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --linux_subsystems DAMON --year 2024 [...] # 20 authors, 157 commits in total $ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --linux_subsystems DAMON --year 2023 [...] # 25 authors, 180 commits in total $ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --linux_subsystems DAMON --year 2022 [...] # 32 authors, 210 commits in total $ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --linux_subsystems DAMON --year 2021 [...] # 11 authors, 75 commits in total 2022 was the busiest year where 32 developers made 210 commits. Compared to the year, the increase of the number of developers (3.12 %) is trivial. But the increase of the number of commits (85.71 %) is still impressive.\nAgain, huge appreciation to all grateful developers!\nDAMON User-space Tool Development of the DAMON user-space tool was also busier than 2024. 12 people made 1,309 commits in 2025. In 2024, 8 people made 1,031 commits. Note that the 2024 statistic counts me twice. Again, huge appreciation to all grateful developers!\n$ ./lazybox/version_control/authors.py ./damo --year 2025 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 1291 commits 2. Bijan Tabatabai \u0026lt;bijantabatab@micron.com\u0026gt;: 3 commits 3. Usama Arif \u0026lt;usamaarif642@gmail.com\u0026gt;: 3 commits 4. Andrew Paniakin \u0026lt;apanyaki@amazon.com\u0026gt;: 2 commits 5. Bijan Tabatabai \u0026lt;btabatabai@wisc.edu\u0026gt;: 2 commits 6. Wu Cheng Han \u0026lt;hank20010209@gmail.com\u0026gt;: 2 commits 7. Michel Lind \u0026lt;salimma@fedoraproject.org\u0026gt;: 1 commits 8. SeungSu Baek \u0026lt;ssbaek@korea.ac.kr\u0026gt;: 1 commits 9. Bijan Tabatabai \u0026lt;bijan311@gmail.com\u0026gt;: 1 commits 10. Nhat Pham \u0026lt;nphamcs@gmail.com\u0026gt;: 1 commits 11. wangchuanguo \u0026lt;wangchuanguo@inspur.com\u0026gt;: 1 commits 12. Akinobu Mita \u0026lt;akinobu.mita@gmail.com\u0026gt;: 1 commits # 12 authors, 1309 commits in total $ ./lazybox/version_control/authors.py ./damo --year 2024 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 753 commits 2. SeongJae Park \u0026lt;sj38.park@gmail.com\u0026gt;: 264 commits 3. Mithun Veluri \u0026lt;velurimithun38@gmail.com\u0026gt;: 4 commits 4. Honggyu Kim \u0026lt;honggyu.kim@sk.com\u0026gt;: 3 commits 5. m8 \u0026lt;umusasadik@gmail.com\u0026gt;: 2 commits 6. Yunjeong Mun \u0026lt;yunjeong.mun@sk.com\u0026gt;: 2 commits 7. Akinobu Mita \u0026lt;akinobu.mita@gmail.com\u0026gt;: 1 commits 8. Alex Rusuf \u0026lt;yorha.op@gmail.com\u0026gt;: 1 commits 9. Piyush Thange \u0026lt;pthange19@gmail.com\u0026gt;: 1 commits # 9 authors, 1031 commits in total Paper Citations I published two academic papers introducing DAMON in 2019 [14] and 2022 [15]. The number of citations for the two papers continued its exponential increase, as below.\n2019-published paper: 10 (2024) -\u0026gt; 18 (2025) 2022-published paper: 7 (2024) -\u0026gt; 14 (2025) Conclusion So, 2025 was the busiest year of DAMON development in its history. That was motivated by automation, observability, and memory tiering needs. Meta, Micron, Huawei, SK hynix and individuals made the major selfish ;) and grateful contributions.\nReferences [1] 2022 retro: https://lore.kernel.org/20221229171209.162356-1-sj@kernel.org/\n[2] 2023 retro: https://lore.kernel.org/20231231222250.140364-1-sj@kernel.org/\n[3] 2024 retro: https://lore.kernel.org/20260216210625.68098-1-sj@kernel.org/\n[4] 2025 DAMON news: https://damonitor.github.io/posts/damon_news/#2025\n[5] OpenSUSE DAMON enable news: https://social.kernel.org/notice/AtQ94OoroZhOGuGuAq\n[6] DAMON enablement: https://lore.kernel.org/all/20250610173228.49109-1-sj@kernel.org/\n[7] DAMON enablement revert: https://git.kernel.org/torvalds/c/aef17cb3d3c438540\n[8] DAMON_STAT enablement on Debian: https://salsa.debian.org/kernel-team/linux/-/merge_requests/1616\n[9] 2025 DAMON features list: https://github.com/damonitor/damo/blob/next/src/_damon_features.py#L235\n[10] page-level monitoring: https://git.kernel.org/torvalds/c/626ffabe67c2359f3\n[11] hugepage monitoring: https://git.kernel.org/torvalds/c/0431c42622612a96cce\n[12] LRU-active monitoring: https://git.kernel.org/torvalds/c/3b23a44f1f1967415\n[13] intervals autotune: https://git.kernel.org/torvalds/c/1eb3471bf5749ff3769e\n[14] memory tiering automation: https://git.kernel.org/torvalds/c/0e1c773b501f3\n[15] memory interleaving: https://git.kernel.org/torvalds/c/a2c24eae5a15f79673e\n[16] damon_stat: https://git.kernel.org/torvalds/c/369c415e60732b7c8ed333688915\n[17] arm32 lpae: https://git.kernel.org/torvalds/c/09a616cbb371e6b843e536f00e38\n[18] hugepage on vaddr: https://git.kernel.org/torvalds/c/408b299a62ec207fa5f21\n[19] memcg tiering: https://git.kernel.org/torvalds/c/d3946c5f4c1c5db63532eb433\n[20] THP use case at OSSummit EU: https://sched.co/25Vrh\n[21] Ravi at hotness meeting: https://lore.kernel.org/all/d952a84f-332e-8f7a-4816-2c1cbd8f5b00@google.com/\n[22] Talk at FOSDEM: https://archive.fosdem.org/2025/schedule/event/fosdem-2025-4396-damon-kernel-subsystem-for-data-access-monitoring-and-access-aware-system-operations/\n[23] Sessions at LSF/MM/BPF: https://lwn.net/Articles/1016525/\n[24] Talk at OSSummit NA: https://ossna2024.sched.com/event/1aBOg\n[25] Talk at Kernel Recipes: https://kernel-recipes.org/en/2025/schedule/overcoming-observer-effects-in-memory-management-with-damon/\n[26] Session1 at LPC: https://lpc.events/event/19/contributions/2066/\n[27] Session2 at LPC: https://lpc.events/event/19/contributions/2059/\n[28] Talk at LPC: https://lpc.events/event/19/contributions/2075/\n[29] authors.py: https://github.com/sjp38/lazybox/blob/master/version_control/authors.py\n","permalink":"https://damonitor.github.io/posts/yearly_retro_2025/","summary":"This is also posted to DAMON mailing list.\nIt is a bit late, but let me share my retrospect of DAMON development for 2025, before my memory goes away. The yearly retrospects for 2022 [1], 2023 [2] and 2024 [3] are also available.\nSummary 2025 was the busiest year for DAMON development of its history. 33 people made 390 commits for DAMON in 2025. Those numbers are 65 % and 157 % increase from those of 2024.","title":"DAMON Yearly Retrospect (2025)"},{"content":"This is also posted to DAMON mailing list.\nAfter DAMON has upstreamed in 2021, I shared yearly retrospects for 2022 [1] and 2023 [2]. And somehow I forgot to do that for 2024 and 2025. Let me share the retrospect of 2024, before my memory goes away.\nSummary 2024 was a year that DAMON has been more publicly and widely adopted. AWS published their usage of DAMON as a VLDB paper. SK hynix developed and upstreamed DAMON features for memory tiering. Debian kernel team has enabled DAMON build config on their kernels. Citations of DAMON papers have also increased from 10 in 2023 to 17 in 2024.\nDAMON\u0026rsquo;s status and future plans have presented and discussed on multiple conferences, including OSSummit NA, LSF/MM/BPF, OSSummit EU, and LPC.\nNumber of development participants and commits have reduced compared to 2023, though. Hopefully that reflects how DAMON continues being stabilized.\nEvents In January, Linux 6.8-rc1 has released, with feedback-driven DAMOS aggressiveness auto-tuning [3] feature. The feature allows users simply feed DAMOS rewards for current aggressiveness, and DAMOS can automatically tune the aggressiveness to convince the user.\nIn March, Linux 6.9-rc1 has released, with DAMOS aggressiveness self-tuning [4] feature. It makes the aggressiveness auto-tuning can work itself, without user\u0026rsquo;s feedback.\nIn April, DAMON was presented [4] at OSSummit North America. The presentation was focusing on the feedback-driven DAMOS auto/self-tuning.\nIn May, DAMON updates and future plans were discussed [5] at LSF/MM/BPF. The plan was including DAMON-based memory tiering and holistic access-aware memory autoscaling for cloud providers.\nIn July, Linux 6.11-rc1 has released, with DAMON changes [6] for CXL memory tiering. The change is developed by SK hynix for their heterogeneous memory software development kit.\nIn July, AWS published a paper [7] introducing their DAMON usage as a part of it, at VLDB. The paper introduces AWS Aurora Serverless V2, which uses DAMON as its core part of memory auto-scaling. Specifically, they use DAMON for proactively reclaiming cold pages inside guests.\nIn August, DAMON project repositories on GitHub has moved [8] from awslabs organization to damonitor organization.\nIn September, DAMON use cases were presented [9] at OSSummit EU. The talk was mainly covering DAMON usages from AWS and SK hynix. Honggyu Kim, an SK hynix engineer who developed DAMON-based memory tiering, has shared their use case on the talk.\nIn September, DAMON long term development plan was presented [10] at LPC Kernel Memory Management Micro-conference. The plan was about how DAMON will be flexible but also just works.\nIn September, Debian enabled [11] DAMON build configuration on their kernels.\nIn December, DAMON monitoring intervals tuning guide has published [12]. The guide includes the monitoring and tuning results on real world server workloads.\nFeatures In 2024, a few new DAMON features have landed on the mainline. Among those, below two features are noticeable to me.\nDAMOS aggressiveness auto-tuning feature have landed in two parts. The first part [3] on v6.8, and the second part [4] on v6.9. This feature makes access aware system operations nearly self-driving.\nDAMON changes for CXL memory tiering has developed by SK hynix and landed [6] on v6.11. The feature allows users to setup DAMOS for migrating hot and cold pages for appropriate tiers.\nStatistics In 2024, 20 people made 157 commits for DAMON on the mainline. I got the numbers with name of the contributors using my script [13] as below.\n$ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --since 2024-01-01 --until 2024-12-31 --linux_subsystems DAMON 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 133 commits 2. Zheng Yejian \u0026lt;zhengyejian@huaweicloud.com\u0026gt;: 2 commits 3. Anna-Maria Behnsen \u0026lt;anna-maria@linutronix.de\u0026gt;: 2 commits 4. Honggyu Kim \u0026lt;honggyu.kim@sk.com\u0026gt;: 2 commits 5. Hyeongtak Ji \u0026lt;hyeongtak.ji@sk.com\u0026gt;: 2 commits 6. Barry Song \u0026lt;v-songbaohua@oppo.com\u0026gt;: 2 commits 7. Akinobu Mita \u0026lt;akinobu.mita@gmail.com\u0026gt;: 1 commits 8. Maximilian Heyne \u0026lt;mheyne@amazon.de\u0026gt;: 1 commits 9. Andrew Paniakin \u0026lt;apanyaki@amazon.com\u0026gt;: 1 commits 10. James Houghton \u0026lt;jthoughton@google.com\u0026gt;: 1 commits 11. Ba Jing \u0026lt;bajing@cmss.chinamobile.com\u0026gt;: 1 commits 12. Leo Stone \u0026lt;leocstone@gmail.com\u0026gt;: 1 commits 13. Jinjie Ruan \u0026lt;ruanjinjie@huawei.com\u0026gt;: 1 commits 14. Diederik de Haas \u0026lt;didi.debian@cknow.org\u0026gt;: 1 commits 15. Liam R. Howlett \u0026lt;Liam.Howlett@oracle.com\u0026gt;: 1 commits 16. Peng Hao \u0026lt;flyingpeng@tencent.com\u0026gt;: 1 commits 17. Christophe Leroy \u0026lt;christophe.leroy@csgroup.eu\u0026gt;: 1 commits 18. Alex Rusuf \u0026lt;yorha.op@gmail.com\u0026gt;: 1 commits 19. Javier Carrasco \u0026lt;javier.carrasco.cruz@gmail.com\u0026gt;: 1 commits 20. Vincenzo Mezzela \u0026lt;vincenzo.mezzela@gmail.com\u0026gt;: 1 commits # 20 authors, 157 commits in total Huge appreciation to all the developers. It was a grateful year to work with you.\nThe last line of the output for 2023 was like below:\n$ ./lazybox/version_control/authors.py ./linux --skip_merge_commits \\ --since 2023-01-01 --until 2023-12-31 --linux_subsystems DAMON [...] # 25 authors, 180 commits in total So, both the number of developers (25 -\u0026gt; 20) and commits (180 -\u0026gt; 157) have reduced. Hopefully this means that DAMON is becoming more stabilized and reliable.\nDAMON User-space Tool DAMON user-space tool is also an important part of DAMON project. In 2024, eight people made 1,031 commits for DAMON user-space tool. The list of the developers are as below:\n$ ./lazybox/version_control/authors.py ./damo --skip_merge_commits \\ --since 2024-01-01 --until 2024-12-31 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 753 commits 2. SeongJae Park \u0026lt;sj38.park@gmail.com\u0026gt;: 264 commits 3. Mithun Veluri \u0026lt;velurimithun38@gmail.com\u0026gt;: 4 commits 4. Honggyu Kim \u0026lt;honggyu.kim@sk.com\u0026gt;: 3 commits 5. m8 \u0026lt;umusasadik@gmail.com\u0026gt;: 2 commits 6. Yunjeong Mun \u0026lt;yunjeong.mun@sk.com\u0026gt;: 2 commits 7. Akinobu Mita \u0026lt;akinobu.mita@gmail.com\u0026gt;: 1 commits 8. Alex Rusuf \u0026lt;yorha.op@gmail.com\u0026gt;: 1 commits 9. Piyush Thange \u0026lt;pthange19@gmail.com\u0026gt;: 1 commits # 9 authors, 1031 commits in total Note that I was counted twice due to different email address usages.\nThe tool\u0026rsquo;s output for 2023 was like below:\n$ ./lazybox/version_control/authors.py ./damo --skip_merge_commits \\ --since 2023-01-01 --until 2023-12-31 1. SeongJae Park \u0026lt;sj38.park@gmail.com\u0026gt;: 1793 commits 2. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 58 commits 3. Honggyu Kim \u0026lt;honggyu.kp@gmail.com\u0026gt;: 8 commits 4. sjpark \u0026lt;sjpark@amazon.com\u0026gt;: 5 commits 5. Michel Alexandre Salim \u0026lt;salimma@fedoraproject.org\u0026gt;: 4 commits 6. SeongJae Park \u0026lt;sjpark@amazon.com\u0026gt;: 3 commits 7. Honggyu Kim \u0026lt;honggyu.kim@sk.com\u0026gt;: 2 commits 8. Andrew Paniakin \u0026lt;apanyaki@amazon.com\u0026gt;: 1 commits 9. fdu \u0026lt;1050329+fdu@users.noreply.github.com\u0026gt;: 1 commits 10. Puranjay Mohan \u0026lt;pjy@amazon.com\u0026gt;: 1 commits 11. Puranjay Mohan \u0026lt;pjy@amazon.de\u0026gt;: 1 commits # 11 authors, 1877 commits in total Note that I was counted four times, and Puranjay was counted twice. So, seven people made 1,877 commits for DAMON user-space tool in 2023. The number of contributors have increased, while the number of total commits have quite reduced. The reduction of the commits was mainly from my side. I was making ~1,800 commits in 2023, but only ~1,000 commits in 2024. I actually feel DAMON user-space tool is being more stabilized. I also spent quite amount of my time that I used to use for DAMON user-space tool for another tool, hkml [16].\nPaper Citations Yet another interesting statistics about the number of DAMON paper citations. I published two academic papers introducing DAMON in 2019 [14] and 2022 [15].\nThe number of citations for the two papers have quite increased, as below.\n2019-published paper: 6 (2023) -\u0026gt; 10 (2024) 2022-published paper: 4 (2023) -\u0026gt; 7 (2024) Conclusion So, 2024 was a year DAMON has more publicly and widely adopted, and extended itself for more use cases.\nReferences [1] 2022 retrospect: https://lore.kernel.org/20221229171209.162356-1-sj@kernel.org/\n[2] 2023 retrospect: https://lore.kernel.org/20231231222250.140364-1-sj@kernel.org/\n[3] DAMOS auto-tune: https://git.kernel.org/torvalds/c/9294a037c01564786\n[4] DAMOS self-tune: https://git.kernel.org/torvalds/c/78f2f60377ee43b7f\n[5] OSSummit NA talk: https://ossna2024.sched.com/event/1aBOg\n[6] DAMOS memory-tiering: https://git.kernel.org/torvalds/c/a00ce85af2a1be49\n[7] VLDB AWS paper: https://www.amazon.science/publications/resource-management-in-aurora-serverless\n[8] awslabs to damonitor GitHub repos move: https://lore.kernel.org/all/20240813232158.83903-1-sj@kernel.org/\n[9] OSSummit EU talk: https://osseu2024.sched.com/event/1ej2S\n[10] LPC talk: https://lpc.events/event/18/contributions/1768/\n[11] Debian DAMON enablement: https://salsa.debian.org/kernel-team/linux/-/merge_requests/1172\n[12] DAMON tuning guide: https://lore.kernel.org/all/20241202175459.2005526-1-sj@kernel.org/\n[13] https://github.com/sjp38/lazybox/blob/master/version_control/authors.py\n[14] 2019 DAMON paper: https://dl.acm.org/doi/abs/10.1145/3366626.3368125\n[15] 2022 DAMON paper: https://dl.acm.org/doi/abs/10.1145/3502181.3531466\n[16] hkml: https://github.com/sjp38/hackermail\n","permalink":"https://damonitor.github.io/posts/yearly_retro_2024/","summary":"This is also posted to DAMON mailing list.\nAfter DAMON has upstreamed in 2021, I shared yearly retrospects for 2022 [1] and 2023 [2]. And somehow I forgot to do that for 2024 and 2025. Let me share the retrospect of 2024, before my memory goes away.\nSummary 2024 was a year that DAMON has been more publicly and widely adopted. AWS published their usage of DAMON as a VLDB paper.","title":"DAMON Yearly Retrospect (2024)"},{"content":"This is also posted to DAMON mailing list.\nHello,\nLast year around this time, I shared a humble retrospect of DAMON for 2022[1], which was effectively the first year it had after it was merged in the mainline. As one more year has passed, I\u0026rsquo;d like to share that again for the second year of DAMON, with some events and statistics of this year\u0026rsquo;s DAMON development that I was personally interested in. I\u0026rsquo;d like to use this chance to say huge thanks to the community, too.\nSummary 2023 was yet another year of active and healthy DAMON development. DAMON has continued active development but hopefully further stabilized. The growth of the DAMON user-space tool, damo, was noticeable.\nDAMON has been explored and used by more people and products. \u0026lsquo;damo\u0026rsquo; got its 100th GitHub star and has been deployed by seven major Linux distros. Four papers and three articles that explore and/or introduce DAMON have been published. A product that manages tiered memory using DAMON has officially been released.\nA substantial amount of development was also continued. Four major DAMON features have been developed. With a significant amount of new features, damo released its second major version (v2.0).\n24 people contributed their great code to DAMON by making 158 commits merged into the mainline. About 28% of the commits were made by Amazon-external contributors. Among the changes for DAMON\u0026rsquo;s parent subsystem, mm, about 9.35% of commits and 8.63% of changed lines were made for DAMON. About 0.21% of the commits for the whole Linux tree were made for DAMON. Six people contributed 1,872 commits to damo.\nCompared to 2022, the absolute number of contributors and changes for DAMON (kernel-only) has reduced (about 40%), though those relative to the numbers for its parent subsystem, mm, and the entire Linux tree were increased (about 10%) or not changed. Meanwhile, the numbers for damo have impressively increased to about 600% and 180% for contributors and changes, respectively.\nKey Events DAMON user-space tool, damo[2], got significant achievements in 2023. \u0026lsquo;damo\u0026rsquo; has deployed via PyPI since August 2021, and ArchLinux started packaging it in March 2022[3]. In 2023, Fedora has been the second major Linux distro that packaging the tool since May[4]. The tool has been packaged by more distros, and as of this writing, ArchLinux, Debian, Devuan, Fedora, Kali Linux, Raspbian, and Ubuntu are[5] packaging the tool. It also got its 100th release[6] and GitHub star in August.\nA few articles and papers introducing or exploring DAMON have been published. Two LWN articles[7,8] that each briefly introduces a new feature of DAMON as a significant change of a new Linux kernel release, and shares detailed LSFMM DAMON discussion, were published in March and May. A blog post[9] from Hocus that introduces DAMON as a way to reduce memory footprint on virtual machines together with free pages reporting was published in July. Two arXiv papers[10,11] and one SOSP paper[12] exploring DAMON on tiered memory management were published in February, September, and October, respectively. One more paper from Intel that improves DAMON for terabyte-scale memory systems was published by ArXiv in November[13].\nWe shared the development status and plans with other kernel developers at LSFMM[14] and Kernel Summit track of LinuxPlumbers[15] in May and November, respectively. We introduced DAMON at a high level for wider audiences at the Open Source Summit North America[16] and Europe[17] in May and September, respectively. In LinuxPlumbers, we had the second in-person DAMON community meetup[18].\nDAMON community started participating[19] in the stable kernels release candidates testing in August.\nSK Hynix released[28] their second major version of the Heterogeneous Memory Software Development Kit (HMSDK), which uses DAMON for their CXL-based tiered memory management, in December.\nA more exhaustive list of events in 2023 is available on the DAMON news web page[29].\nKey Features We started 2023 with Linux v6.2 which delivered DAMOS tried regions feature[20] that was developed in 2022.\nDAMOS filters[21] feature has been developed and merged in the mainline by v6.3-rc1, which was released in March. The feature was later expanded[22] for more use cases and merged in the mainline by v6.6-rc1, which was released in September.\nTwo more DAMON features for pseudo-moving average-based reliable and speedy DAMON snapshot generation[23] and DAMOS-dedicated apply time interval[24] have been developed and merged in the mainline by v6.7-rc1, which was released in November.\nFinally, we developed user feedback-based DAMOS aggressiveness auto-tuning[25], which is currently merged in the mm tree. Hopefully, it will be merged in the mainline by v6.8-rc1, which is expected to be released in January 2024.\nDevelopment Statistics To appreciate and list all names of people who made DAMON available, and to quantify what 2023 was for DAMON development, I collected some numbers using my humble and buggy scripts. The scripts are available as open source[26,27].\nPlease note that numbers don\u0026rsquo;t say everything. Nonetheless, in my humble opinion, those are better than nothing and fun ;)\nContributors According to the humble script, 24 people have contributed to DAMON development in 2023 (v6.2-rc1..v6.7-rc8).\n$ ./lazybox/git_helpers/authors.py ./linux \\ --commits_range v6.2-rc1..v6.7-rc8 --skip_merge_commits \\ --linux_subsystems \u0026quot;DATA ACCESS MONITOR\u0026quot; 1. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 119 commits 2. Kefeng Wang \u0026lt;wangkefeng.wang@huawei.com\u0026gt;: 10 commits 3. Ryan Roberts \u0026lt;ryan.roberts@arm.com\u0026gt;: 4 commits 4. Jinjie Ruan \u0026lt;ruanjinjie@huawei.com\u0026gt;: 3 commits 5. Bjorn Helgaas \u0026lt;bhelgaas@google.com\u0026gt;: 2 commits 6. Vishal Moola (Oracle) \u0026lt;vishal.moola@gmail.com\u0026gt;: 2 commits 7. Hyeongtak Ji \u0026lt;hyeongtak.ji@sk.com\u0026gt;: 1 commits 8. Dan Carpenter \u0026lt;dan.carpenter@linaro.org\u0026gt;: 1 commits 9. Huan Yang \u0026lt;link@vivo.com\u0026gt;: 1 commits 10. Andreas Gruenbacher \u0026lt;agruenba@redhat.com\u0026gt;: 1 commits 11. Juntong Deng \u0026lt;juntong.deng@outlook.com\u0026gt;: 1 commits 12. Levi Yun \u0026lt;ppbuk5246@gmail.com\u0026gt;: 1 commits 13. Suren Baghdasaryan \u0026lt;surenb@google.com\u0026gt;: 1 commits 14. Feng Tang \u0026lt;feng.tang@intel.com\u0026gt;: 1 commits 15. Hugh Dickins \u0026lt;hughd@google.com\u0026gt;: 1 commits 16. Anders Roxell \u0026lt;anders.roxell@linaro.org\u0026gt;: 1 commits 17. Thomas Weißschuh \u0026lt;linux@weissschuh.ne\u0026gt;: 1 commits 18. andrew.yang \u0026lt;andrew.yang@mediatek.com\u0026gt;: 1 commits 19. Baolin Wang \u0026lt;baolin.wang@linux.alibaba.com\u0026gt;: 1 commits 20. Thomas Weißschuh \u0026lt;linux@weissschuh.net\u0026gt;: 1 commits 21. Liam R. Howlett \u0026lt;Liam.Howlett@oracle.com\u0026gt;: 1 commits 22. Huaisheng Ye \u0026lt;huaisheng.ye@intel.com\u0026gt;: 1 commits 23. Hui Su \u0026lt;suhui_kernel@163.com\u0026gt;: 1 commits 24. Xu Panda \u0026lt;xu.panda@zte.com.cn\u0026gt;: 1 commits # 24 authors, 158 commits in total The last line of the output for 2022[3] was as below.\n# 40 authors, 275 commits in total The number has quite reduced compared to that of 2022 (40 -\u0026gt; 24). That\u0026rsquo;s hopefully because DAMON code has more stabilized.\nFor the DAMON user-space tool, damo, six people contributed 1,859 commits in 2023. Note that there were contributors using multiple email addresses.\n$ ./lazybox/git_helpers/authors.py ./damo --since 2023-01-01 1. SeongJae Park \u0026lt;sj38.park@gmail.com\u0026gt;: 1788 commits 2. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 58 commits 3. Honggyu Kim \u0026lt;honggyu.kp@gmail.com\u0026gt;: 8 commits 4. sjpark \u0026lt;sjpark@amazon.com\u0026gt;: 5 commits 5. Michel Alexandre Salim \u0026lt;salimma@fedoraproject.org\u0026gt;: 4 commits 6. SeongJae Park \u0026lt;sjpark@amazon.com\u0026gt;: 3 commits 7. Honggyu Kim \u0026lt;honggyu.kim@sk.com\u0026gt;: 2 commits 8. Andrew Paniakin \u0026lt;apanyaki@amazon.com\u0026gt;: 1 commits 9. fdu \u0026lt;1050329+fdu@users.noreply.github.com\u0026gt;: 1 commits 10. Puranjay Mohan \u0026lt;pjy@amazon.com\u0026gt;: 1 commits 11. Puranjay Mohan \u0026lt;pjy@amazon.de\u0026gt;: 1 commits # 11 authors, 1872 commits in total The numbers for 2022 are as below.\n$ ./lazybox/git_helpers/authors.py ./damo \\ --since 2022-01-01 --until 2022-12-31 --skip_merge_commits 1. SeongJae Park \u0026lt;sj38.park@gmail.com\u0026gt;: 1021 commits 2. SeongJae Park \u0026lt;sj@kernel.org\u0026gt;: 22 commits 3. SeongJae Park \u0026lt;sjpark@amazon.com\u0026gt;: 10 commits 4. SeongJae Park \u0026lt;sjpark@amazon.de\u0026gt;: 1 commits # 4 authors, 1054 commits in total So the number of commits and the contributors have increased to about 180 percent (1,054 -\u0026gt; 1,872) and 600 percent (1 -\u0026gt; 6) respectively, compared to those of 2022. I\u0026rsquo;d call this a huge increase.\nPlease note that there were many unsung hero contributors gave valuable inputs, discussions, and many more things for DAMON development. So shameful that I cannot put everyone\u0026rsquo;s name here.\nThank you so much to all the awesome contributors!\nContributions from non-maintainer The maintainer, SJ (sj@kernel.org), has driven the development of DAMON, but the help from the community was huge. About 28.31% of DAMON commits have been made by people other than SJ. Again, the number is reduced compared to that of last year (33% -\u0026gt; 28%).\n$ ./damon-hack/stat_damon_portion_community_commits.sh \\ ./linux ./damon-hack/stat_branches_2023 range from_sj non_sj non_sj_portion v6.2-rc1..v6.3 32 16 33.33% v6.3..v6.4 0 5 100.00% v6.4..v6.5 19 8 29.63% v6.5..v6.6 20 7 25.93% v6.6..v6.7-rc8 48 11 18.64% v6.2-rc1..v6.7-rc8 119 47 28.31% The output for 2022 was as below.\nrange from_sj non_sj non_sj_portion v5.15..v5.16 50 13 20.63% v5.16..v5.17 16 10 38.46% v5.17..v5.18 26 10 27.78% v5.18..v5.19 23 7 23.33% v5.19..v6.0 15 14 48.28% v6.0..v6.1 33 36 52.17% v6.1..v6.2-rc1 28 6 17.65% v5.15..v6.2-rc1 191 95 33.22% The Portion of DAMON Commits in MM and Linux To show how much change DAMON made to its parent subsystem, MM, and whole Linux, I counted the number of commits that touched files for DAMON, MM, and the whole Linux tree in 2023 releases (v6.2-rc1..v6.7-rc8).\nAccording to the buggy and humble script, about 9.35% of MM commits, and 0.21% of Linux commits were made for DAMON.\n$ ./damon-hack/stat_damon_portion_nr_commits.sh \\ ./linux ./damon-hack/stat_branches_2023 range damon mm damon/mm linux damon/linux v6.2-rc1..v6.3 47 458 10.26% 16273 0.29% v6.3..v6.4 5 346 1.45% 14835 0.03% v6.4..v6.5 26 327 7.95% 13561 0.19% v6.5..v6.6 25 402 6.22% 14069 0.18% v6.6..v6.7-rc8 55 341 16.13% 17138 0.32% v6.2-rc1..v6.7-rc8 158 1690 9.35% 75876 0.21% The output for 2022 was as below.\nrange damon mm damon/mm linux damon/linux v5.15..v5.16 45 307 14.66% 14190 0.32% v5.16..v5.17 17 223 7.62% 13038 0.13% v5.17..v5.18 29 448 6.47% 14954 0.19% v5.18..v5.19 24 399 6.02% 15134 0.16% v5.19..v6.0 15 283 5.30% 15402 0.10% v6.0..v6.1 61 536 11.38% 13942 0.44% v6.1..v6.2-rc1 20 250 8.00% 13687 0.15% v5.15..v6.2-rc1 211 2446 8.63% 100347 0.21% Compared to the numbers for 2022, absolute number of DAMON commits has reduced (211 -\u0026gt; 158), but the proportion of DAMON commits against MM commits has a bit increased (8.63% -\u0026gt; 9.35%), while the portion against Linux has not changed (0.21%).\nBy Number of Lines I further counted the portion of the number of changed lines. Didn\u0026rsquo;t count that for Linux here due to the slow speed of the script. The script argues about 4.05% of the changed lines for MM subsystem were for DAMON. This is a quite decrease compared to that of last year (14.32%). Hopefully, that\u0026rsquo;s because DAMON became more stabilized.\n$ ./damon-hack/stat_damon_portion_lines.sh \\ ./linux ./damon-hack/stat_branches_2023 range damon mm damon/mm v6.2-rc1..v6.3 962 13852 6.94% v6.3..v6.4 32 14226 0.22% v6.4..v6.5 113 9852 1.15% v6.5..v6.6 322 7862 4.10% v6.6..v6.7-rc8 850 10489 8.10% v6.2-rc1..v6.7-rc8 2279 56281 4.05% The output for 2022 was as below.\nrange damon mm damon/mm v5.15..v5.16 2157 8503 25.37% v5.16..v5.17 324 9370 3.46% v5.17..v5.18 3462 16288 21.25% v5.18..v5.19 929 10185 9.12% v5.19..v6.0 870 8665 10.04% v6.0..v6.1 1752 25844 6.78% v6.1..v6.2-rc1 3309 10544 31.38% v5.15..v6.2-rc1 12803 89399 14.32% Conclusion DAMON community delivered multiple important features and a significant amount of changes to the world via the collaboration between the 24 great contributors. I would call 2023 as yet another successful and grateful years of DAMON development.\nHuge thanks to you again, DAMON community. Looking forward to continuing our journey in 2024.\nHope you all enjoy the remaining holidays and a happy new year!\nThanks, SJ\nReferences [1] 2022 retrospect: https://lore.kernel.org/damon/20221229171209.162356-1-sj@kernel.org/\n[2] DAMON user-space tool, damo: https://github.com/awslabs/damo\n[3] ArchLinux damo package: https://aur.archlinux.org/packages/damo\n[4] Fedora damo package: https://packages.fedoraproject.org/pkgs/python-damo/damo/\n[5] damo packaging status: https://repology.org/project/damo/versions\n[6] 100th damo release: https://lore.kernel.org/damon/20230807202044.98700-1-sj@kernel.org/\n[7] LWN article about DAMOS filter: https://lwn.net/Articles/924384/\n[8] LWN article about DAMON LSFMM discussion: https://lwn.net/Articles/931769/\n[9] Hocus article: https://hocus.dev/blog/qemu-vs-firecracker/\n[10] arXiv paper exploring DAMON: https://arxiv.org/pdf/2302.09468.pdf\n[11] arXiv paper exploring DAMON: https://arxiv.org/pdf/2309.01736.pdf\n[12] SOSP paper exploring DAMON: https://dl.acm.org/doi/10.1145/3600006.3613167\n[13] arXiv paper improving DAMON: https://arxiv.org/pdf/2311.10275.pdf\n[14] LSFMM DAMON talk: https://www.youtube.com/watch?v=bbC23ApPvow\n[15] Ksummit DAMON talk: https://lpc.events/event/17/contributions/1624/\n[16] OSSNA DAMON talk: https://ossna2023.sched.com/event/1K5HS\n[17] OSSEU DAMON talk: https://osseu2023.sched.com/event/1OGf9\n[18] LPC DAMON meetup: https://lpc.events/event/17/contributions/1652/\n[19] DAMON\u0026rsquo;s stable rc kernel test results report: https://lore.kernel.org/damon/20230802173033.108621-1-sj@kernel.org/\n[20] DAMOS tried regions: https://lore.kernel.org/damon/20221101220328.95765-1-sj@kernel.org/\n[21] DAMOS filters: https://lore.kernel.org/damon/20221205230830.144349-1-sj@kernel.org/\n[22] DAMOS filters extension: https://lore.kernel.org/damon/20230802214312.110532-1-sj@kernel.org/\n[23] DAMON pseudo-moving snapshot: https://lore.kernel.org/damon/20230915025251.72816-1-sj@kernel.org/\n[24] DAMOS apply interval: https://lore.kernel.org/damon/20230916020945.47296-1-sj@kernel.org/\n[25] Goal-oriented feedback-driven DAMOS auto-tuning: https://lore.kernel.org/damon/20231130023652.50284-1-sj@kernel.org/\n[26] Statistics tool: https://github.com/sjp38/lazybox\n[27] Statistics tool 2: https://git.kernel.org/sj/damon-hack/h/master\n[28] SK HMSDK v2 release: https://github.com/skhynix/hmsdk/releases/tag/hmsdk-v2.0\n[29] 2023 DAMON News: https://sjp38.github.io/post/damon_news/#2023\n","permalink":"https://damonitor.github.io/posts/yearly_dev_summary_2023/","summary":"This is also posted to DAMON mailing list.\nHello,\nLast year around this time, I shared a humble retrospect of DAMON for 2022[1], which was effectively the first year it had after it was merged in the mainline. As one more year has passed, I\u0026rsquo;d like to share that again for the second year of DAMON, with some events and statistics of this year\u0026rsquo;s DAMON development that I was personally interested in.","title":"Looking back on DAMON development in 2023"},{"content":"DAMON exports data access pattern of the system and workload to the user space in multiple ways. Tracepoint is one such way DAMON is using to provide the full access pattern monitoring results in real time.\nDAMON user-space tool (damo) collects and visualizes the information in a handy and human-friendly way. For the collection, damo was internally using perf. On some environments including Android, perf is not available. This made use of DAMON on Android and similar platforms difficult. Kernel version dependency of perf was also one problem of poor damo user experience. Some people had to make their downstream hacks on damo, or add some workarounds.\nOn 2025-12-29, damo v3.0.9 is released. The version drops the perf dependency by alternatively using trace-cmd if available. Because trace-cmd is more universal than perf and has less dependency to kernel versions, users on Android and a few restricted environments can more easily use DAMON.\n","permalink":"https://damonitor.github.io/posts/damo_support_trace_cmd/","summary":"DAMON exports data access pattern of the system and workload to the user space in multiple ways. Tracepoint is one such way DAMON is using to provide the full access pattern monitoring results in real time.\nDAMON user-space tool (damo) collects and visualizes the information in a handy and human-friendly way. For the collection, damo was internally using perf. On some environments including Android, perf is not available. This made use of DAMON on Android and similar platforms difficult.","title":"damo supports recording data access patterns on Android"},{"content":"A number of DAMON changes for Linux 6.19-rc1 has merged into the mainline as a part of MM pull reqeust for Linux 6.19-rc1, which was sent by Andrew Morton. To list the changes here again:\nmm/damon: allow DAMOS auto-tuned for per-memcg per-node memory usage (patches) mm/damon: fixes for address alignment issues in DAMON_LRU_SORT and DAMON_RECLAIM (patches) mm/damon: misc documentation fixups (patches) mm/damon: support pin-point targets removal (patches) mm/damon/tests: fix memory bugs in kunit tests (patches) mm/damon/tests: add more tests for online parameters commit (patches) mm/damon: misc cleanups (patches) ","permalink":"https://damonitor.github.io/posts/damon_changes_for_linux_6.19/","summary":"A number of DAMON changes for Linux 6.19-rc1 has merged into the mainline as a part of MM pull reqeust for Linux 6.19-rc1, which was sent by Andrew Morton. To list the changes here again:\nmm/damon: allow DAMOS auto-tuned for per-memcg per-node memory usage (patches) mm/damon: fixes for address alignment issues in DAMON_LRU_SORT and DAMON_RECLAIM (patches) mm/damon: misc documentation fixups (patches) mm/damon: support pin-point targets removal (patches) mm/damon/tests: fix memory bugs in kunit tests (patches) mm/damon/tests: add more tests for online parameters commit (patches) mm/damon: misc cleanups (patches) ","title":"Major DAMON changes that merged for Linux 6.19"},{"content":"Updates after initial posting.\n2026-02-18 update: LSF/MM/BPF topic for discussing the NUMA hinting faults reuse is posted.\n2026-01-13 update: The RFC v3 has posted to the mailing list.\nBelow is also sent as a mail to DAMON mailing list and relevant people.\nI\u0026rsquo;m working [1] on extending DAMON to monitor accesses that are made by specific CPUs, and/or for writes. The aimed usages include NUMA hit/miss monitoring [2], Kernel Same page Merging scan target selection [3,4], cache aware CPU scheduling, live migration target VM selection [5], and general NUMA-aware pages migration [6].\nI posted its first working version as RFC v2 [7] about four months ago. After the posting, it got interests and tests publicly [5] and privately. RFC v2 was, however, a never upstreamable hack that was made for only showing if it can be implemented. For making it upstreamed, a significant amount of efforts are expected [8]. And after the posting, I was unfortunately unable to spend meaningful time on it due to other events and work. I received more continued interests in the work recently, and hence I\u0026rsquo;m trying to spend more time on it.\nThe Rough Plan The rough plan of it that I shared on DAMON blog [8] was just saying \u0026ldquo;it needs to be upstreamed and it will require a significant amount of effort\u0026rdquo;. So I\u0026rsquo;m sharing yet another rough plan here.\nThis work require multiple changes including\nextending DAMON for receiving access report from other subsystems (damon_report_access() [9]), making other subsystem who can capture origin CPU/writeness of accesses (e.g., page fault handler [10]) report the information to DAMON, extending DAMON for filtering the reports for origin CPU/writeness of accesses. The first part in the current implementation has many rooms to improve in terms of performance. I don\u0026rsquo;t think that\u0026rsquo;s upstreaming blocker though, unless some early testers find a problem of it. So I think this is not a blocker for this project, at least at the moment.\nFor the second part, I\u0026rsquo;m trying to modify the page fault handler, to use information like NUMA balancing hint faults. As Lorenzo pointed out on RFC v2, it needs to have clean and non disruptive design. Also this requires alignment of multiple stakeholders, including MM core and NUMA balancing maintainers. I think this is the biggest challenge of this project. Using other subsystems or features such as instruction sampling can be another way to go. But for general availability of it on any architecture and virtual machine internal cases, people I privately discussed preferred this fault based approach. I will try to make the alignment with the stakeholders, after the upcoming LPC [11]. Unless I make good progress on this part very quickly, I will propose an LSFMMBPF'26 [12] session for this. So hopefully we will make some alignment among the stakeholders by LSFMM. Hopefully we will get a more concrete upstreaming plan and timeline, at least after the LSFMM'26.\nThe third part in current implementation is very ugly. Especially its kernel API and user ABI need to be redesigned from the beginning. The interface is better to be stabilized sooner, as changes of the interface will make the early users/testers be confused and frustrated. I started working on this, and hopefully I will share RFC v3 of this project with the stabilized interface, by upcoming LPC'25.\nSummary and Message for User/Tester To summarize the major expected timeline here,\nBy LPC'25 (2025-12-13), another version having stabilized DAMON interface will be shared. By LSFMMBPF'26 (2026-05-06), an alignment on part 2 stakeholders will be made. After the alignment, a more clear plan will be made. If you are already using or testing RFC v2 of this project, please be aware the kernel API and user ABI will significantly be changed on the version that I am willing to share by LPC'25. It means if you are directly using or testing the features via the kernel API and/or user ABI, your usages or tests might need to be updated.\nThe support of the features on DAMON user-sapce tool (damo) is also experimental at the moment, and that will also be changed in more stable form in future. But, the experimental interface will support both before-LPC and after-LPC DAMON for the short term. Hence, if you are using or testing the features via DAMON user-space tool\u0026rsquo;s experimental interface, you will not need to make any change in the short term.\nPlease feel free to raise any questions or comments on the plan and the project.\nReferences [1] https://damonitor.github.io/posts/write_only_cpus_only_monitoring/\n[2] https://lore.kernel.org/linux-mm/cover.1645024354.git.xhao@linux.alibaba.com/\n[3] https://lore.kernel.org/lkml/20220203131237.298090-1-pedrodemargomes@gmail.com/\n[4] https://lore.kernel.org/damon/20250129044041.25884-1-pedrodemargomes@gmail.com/\n[5] https://lore.kernel.org/all/aJAfTUh-49pYuhbg@3c06303d853a/\n[6] https://lpc.events/event/19/contributions/2066/\n[7] https://lore.kernel.org/all/20250727201813.53858-1-sj@kernel.org/\n[8] https://damonitor.github.io/posts/write_only_cpus_only_monitoring/#cautions-and-plan-to-drop-experimental-tag\n[9] https://lwn.net/Articles/1016525/\n[10] https://lore.kernel.org/all/20250727201813.53858-6-sj@kernel.org/\n[11] https://lpc.events/event/19/\n[12] https://events.linuxfoundation.org/lsfmmbpf/\n","permalink":"https://damonitor.github.io/posts/damon_cpus_write_monitoring_rfc_v3_plan/","summary":"Updates after initial posting.\n2026-02-18 update: LSF/MM/BPF topic for discussing the NUMA hinting faults reuse is posted.\n2026-01-13 update: The RFC v3 has posted to the mailing list.\nBelow is also sent as a mail to DAMON mailing list and relevant people.\nI\u0026rsquo;m working [1] on extending DAMON to monitor accesses that are made by specific CPUs, and/or for writes. The aimed usages include NUMA hit/miss monitoring [2], Kernel Same page Merging scan target selection [3,4], cache aware CPU scheduling, live migration target VM selection [5], and general NUMA-aware pages migration [6].","title":"A rough plan for CPUs/write-only monitoring RFC v3 and future"},{"content":"This post lists research papers and news articles citing DAMON in an interesting way rather than just simple name listing. The list is collected in a human sense from incomplete searching, so it is quite far from perfect. Please reach out to sj@kernel.org if you want updates on the list.\nYet another way to get the list would be using the Google Scholar for DAMON author-published paper ciataions (1, 2).\n2026 Hamid Hadian, Jinshu Liu*, Hanchen Xu*, Hansen Idden, Huaicheng Li, \u0026ldquo;PACT: A Criticality-First Design for Tiered Memory.\u0026rdquo; ASPLOS 2026, https://huaicheng.github.io/p/asplos26-pact.pdf Kanellis, Konstantinos, et al. \u0026ldquo;From Good to Great: Parameter Tuning in Memory Tiering Systems.\u0026rdquo; IEEE Transactions on Computers (2026). DOI: 10.1109/TC.2026.3656472 https://ieeexplore.ieee.org/document/11359582 Tong Xing, Jiaxun Yang, Javier Picorel, and Antonio Barbalace. 2026. Rethinking Tiered Memory Management in Cloud Data Centers. In Proceedings of the 2025 ACM Symposium on Cloud Computing (SoCC \u0026lsquo;25). Association for Computing Machinery, New York, NY, USA, 74–87. https://doi.org/10.1145/3772052.3772213 2025 Junliang Hu, Zhisheng Hu, Chun-Feng Wu, and Ming-Chang Yang. 2025. Demeter: A Scalable and Elastic Tiered Memory Solution for Virtualized Cloud via Guest Delegation. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP \u0026lsquo;25). Association for Computing Machinery, New York, NY, USA, 169–185. https://doi.org/10.1145/3731569.3764801 Taehyung Lee, Sumit Monga, Young Ik Eom, and Changwoo Min. 2025. Harnessing Page Access Frequency Distribution for Efficient Memory Tiering. ACM Trans. Comput. Syst. Just Accepted (November 2025). https://doi.org/10.1145/3777549 Baumstark, Alexander; Paradies, Marcus; Sattler, Kai-Uwe (2025): Lightweight Memory Access Monitoring for Dynamic Data Placement in Tiered Memory Systems. Datenbanksysteme für Business, Technologie und Web (BTW 2025). DOI: 10.18420/BTW2025-12. Gesellschaft für Informatik, Bonn. EISSN: 2944-7682. pp. 265-276. Research Track. Bamberg. 3.-7. März 2025 Kavya Shekar and Dan Williams. 2025. SchedBPF - Scheduling BPF programs. In Proceedings of the 3rd Workshop on eBPF and Kernel Extensions (eBPF \u0026lsquo;25). Association for Computing Machinery, New York, NY, USA, 45–47. https://doi.org/10.1145/3748355.3748366 Wanju Doh, Yaebin Moon, Seoyoung Ko, Seunghwan Chung, Kwanhee Kyung, Eojin Lee, and Jung Ho Ahn. 2025. PET: Proactive Demotion for Efficient Tiered Memory Management. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys \u0026lsquo;25). Association for Computing Machinery, New York, NY, USA, 854–869. https://doi.org/10.1145/3689031.3717471 Fisher, Max Henry. Analysis of Memory Access Patterns for Large Language Model Inference. Diss. Virginia Tech, 2025. https://hdl.handle.net/10919/135946 2024 Yuhong Zhong, Daniel S. Berger, Carl Waldspurger, Ryan Wee, Ishwar Agarwal, Rajat Agarwal, Frank Hady, Karthik Kumar, Mark D. Hill, Mosharaf Chowdhury, and Asaf Cidon. 2024. Managing memory tiers with CXL in virtualized environments. In Proceedings of the 18th USENIX Conference on Operating Systems Design and Implementation (OSDI'24). USENIX Association, USA, Article 3, 37–56. https://www.usenix.org/conference/osdi24/presentation/zhong-yuhong\nJie Ren, Dong Xu, Junhee Ryu, Kwangsik Shin, Daewoo Kim, and Dong Li. 2024. MTM: Rethinking Memory Profiling and Migration for Multi-Tiered Large Memory. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys \u0026lsquo;24). Association for Computing Machinery, New York, NY, USA, 803–817. https://doi.org/10.1145/3627703.3650075\nNair, Alan, SANDEEP KUMAR, and ARAVINDA PRASAD. \u0026ldquo;Telescope: Profiling Memory Access Patterns at the Terabyte-scale.\u0026rdquo; 2024 USENIX Annual Technical Conference (USENIX ATC 24). 2024.\nJuneseo Chang, Wanju Doh, Yaebin Moon, Eojin Lee, and Jung Ho Ahn. 2024. IDT: Intelligent Data Placement for Multi-tiered Main Memory with Reinforcement Learning. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (HPDC \u0026lsquo;24). Association for Computing Machinery, New York, NY, USA, 69–82. https://doi.org/10.1145/3625549.3658659 2023 ====\nSai Sha, Chuandong Li, Yingwei Luo, Xiaolin Wang, and Zhenlin Wang. 2023. VTMM: Tiered Memory Management for Virtual Machines. In Proceedings of the Eighteenth European Conference on Computer Systems (EuroSys \u0026lsquo;23). Association for Computing Machinery, New York, NY, USA, 283–297. https://doi.org/10.1145/3552326.3587449\n","permalink":"https://damonitor.github.io/posts/citations/","summary":"This post lists research papers and news articles citing DAMON in an interesting way rather than just simple name listing. The list is collected in a human sense from incomplete searching, so it is quite far from perfect. Please reach out to sj@kernel.org if you want updates on the list.\nYet another way to get the list would be using the Google Scholar for DAMON author-published paper ciataions (1, 2).","title":"Citations"},{"content":"2025-12-08 update: The RFC v3 is posted.\nFrom the very early days of DAMON, there were attempts to extend it for cpus-aware monitoring and write-only monitoring.\nIn 2022, Xin Hao proposed extending DAMON for NUMA access statistics.\nIn 2022 and 2025, Pedro Demarchi Gomes proposed extending DAMON for writes-only monitoring.\nThose proposals are not yet upstreamed though. We continued similar DAMON extension discussions publicly and privately, with multiple parties, though.\nI was recently taking some time on these, and happy to announce the first working prototype for the extensions. Please note that it is working in a way that you can at least try, but the implementation is gross, unupstreamable, and may contain many bugs.\nOn Linux kernel that built with some hacks that exist in damon/next tree as of this writing (2025-08-03), users can do data access monitoring for only writes, or for accesses made by given set of CPUs using the latest version of DAMON user-space tool, damo, on its development branch.\nWrite-only Access Monitoring Users can do write-only data access monitoring by adding --exp_ops_use_reports and --exp_ops_write_only options with value y to the usual DAMON parameters setup commands including damo start and damo tune, like below.\ndamo start --exp_ops_use_reports y --exp_ops_write_only y Then DAMON will silently ignore read accesses, and show only the accesses that made for writes.\nNote that only physical address space monitoring is supported for now.\nCPUs-only Access Monitoring Users can monitor the data accesses made by only specific CPUs by adding --exp_ops_use_reports and --exp_ops_cpus options to the usual DAMON parameters setup commands including damo start and damo tune, like below.\ndamo start --exp_ops_use_reports y --exp_ops_cpus 0-3,5-7,9 --exp_ops_use_reports argument should always be y.\n--exp_ops_cpus option should be given with a list of the desired CPUs. In this example, accesses that made by only CPUs of id 0, 1, 2, 3, 5, 6, 7, or 9 will be monitored and reported. The format of the cpus list is same to that for cpuset.cpus file of cgroup v2.\nNote that only physical address space monitoring is supported for now.\nCautions and Plan to Drop Experimental Tag The write-only and cpus-only monitoring features require changes in Linux kernel that not yet upstreamed. The changes are available at damon/next tree, which is for containing under-development DAMON works, as of this writing (2025-08-03). To make those upstreamed, we may need significant amount of efforts. Many changes could be made on the upstreamed version. In the worst case, we would forgive upstreaming the changes to mainline kernel, and drop the support on the damon/next tree.\nThe user interface for these features on damo will definitely be changed in future. At least, --exp_ part will be removed.\nFinally, as of this writing (2025-08-03), the features are nearly not tested, so there could be many bugs.\nSo, please feel free to try and let us know bugs that you found, and if it is useful and worthy enough to make the upstream efforts. But, please don\u0026rsquo;t make your long term important works depend on these.\n","permalink":"https://damonitor.github.io/posts/write_only_cpus_only_monitoring/","summary":"2025-12-08 update: The RFC v3 is posted.\nFrom the very early days of DAMON, there were attempts to extend it for cpus-aware monitoring and write-only monitoring.\nIn 2022, Xin Hao proposed extending DAMON for NUMA access statistics.\nIn 2022 and 2025, Pedro Demarchi Gomes proposed extending DAMON for writes-only monitoring.\nThose proposals are not yet upstreamed though. We continued similar DAMON extension discussions publicly and privately, with multiple parties, though.","title":"DAMON for Write-only or Given CPUs-only Monitoring"},{"content":"TL; DR: damo report heatmap has recently advanced to support modern DAMON features including age tracking, snapshots, page level monitoring, and monitoring intervals auto-tuning. It will help users intuitively understand the monitored access patterns at a glance.\nDAMON in The Past: Full Recording based Monitoring At the beginning, DAMON was providing only the access frequency of each memory region in real time. Hence heatmap visualization, which shows the access frequency of each memory area in the timeline was the first and one of the best ways to see the access pattern. DAMON user-space tool (damo) supported such collections and visualizations of the data via damo record and damo report heatmap, like below example.\n$ sudo damo record \u0026#34;../masim/masim ../masim/configs/stairs.cfg\u0026#34; [...] $ sudo damo report heatmap -i damon.data 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000004777777600000 00000000000000000000000000000000000000000000000000000000000000000005888888700000 00000000000000000000000000000000000000000000000000000000000000000003555555400000 00000000000000000000000000000000000000000000000000000000000048888883000000000000 00000000000000000000000000000000000000000000000000000000000068888883000000000000 00000000000000000000000000000000000000000000000000000000000068888883000000000000 00000000000000000000000000000000000000000000000000000111111100000000000000000000 00000000000000000000000000000000000000000000000000004888888700000000000000000000 00000000000000000000000000000000000000000000000000005888888700000000000000000000 00000000000000000000000000000000000000000000000000001222222100000000000000000000 00000000000000000000000000000000000000000000088888883000000000000000000000000000 00000000000000000000000000000000000000000000088888883000000000000000000000000000 00000000000000000000000000000000000000000000056666662000000000000000000000000000 00000000000000000000000000000000000001777776600000000000000000000000000000000000 00000000000000000000000000000000000004888888700000000000000000000000000000000000 00000000000000000000000000000000000004888888800000000000000000000000000000000000 00000000000000000000000000000023333331000000000000000000000000000000000000000000 00000000000000000000000000000088888883000000000000000000000000000000000000000000 00000000000000000000000000000588888883000000000000000000000000000000000000000000 00000000000000000000011111111332222221000000000000000000000000000000000000000000 00000000000000000000038888888870000000000000000000000000000000000000000000000000 00000000000000000000038888888870000000000000000000000000000000000000000000000000 00000000000000000000037666666650000000000000000000000000000000000000000000000000 00000000000002888888888000000000000000000000000000000000000000000000000000000000 00000000000003888888888000000000000000000000000000000000000000000000000000000000 00000000000003888888888000000000000000000000000000000000000000000000000000000000 00000004444555610000000000000000000000000000000000000000000000000000000000000000 00000005888888820000000000000000000000000000000000000000000000000000000000000000 00000005888888820000000000000000000000000000000000000000000000000000000000000000 44444446722222200000000000000000000000000000000000000000000000000000000000000000 88888888800000000000000000000000000000000000000000000000000000000000000000000000 88888888800000000000000000000000000000000000000000000000000000000000000000000000 66666666600000000000000000000000000000000000000000000000000000000000000000000000 # access_frequency: 0123456789 # x-axis: space [127.934 TiB, 127.934 TiB) (101.930 MiB) # y-axis: time [2 h 33 m 56.451 s, 2 h 34 m 56.333 s) (59.882 s) # resolution: 80x40 (1.274 MiB and 1.497 s for each character) Each character on the middle of the output shows when (row) what address range (column) of the memory was how frequently (number) accessed. masim with stairs.cfg allocated 10 regions of 10 MiB size, and accesses those one by one. The above heatmap is hence showing the pattern.\nFor the visualization, however, users have to record the entire DAMON-observed access frequency of each region for every moment. For long time recording, storage usage of the recorded data was not negligible. Heatmap-visualization of the huge record data was also time consuming. Hence this was useful for lab environments, but arguably not optimized for production environments.\n$ ls -alh damon.data -rw------- 1 root root 95K Jun 8 12:11 damon.data Evolvement of Snapshots-based Monitoring DAMON has evolved to provide not only access frequency of each region, but also how long the current access frequency of the region was kept, namely \u0026lsquo;age\u0026rsquo;. It was mainly developed for access-aware system operations, namely DAMON-based Operation Schemes (DAMOS). But, we found the information can also be useful for lightweight but practical monitoring. We therefore made yet another DAMON feature for getting only the current snapshot of the DAMON monitoring results. For easy capturing and visualization of the snapshot data, we implemented yet another user-space tool feature, namely damo report access. damo record, which is the user-space tool feature for capturing entire DAMON monitoring results for the heatmap-like visualizations, has also been updated to support only snapshot capturing (--snapshot option). Nowadays, the snapshot based visualization is the main feature of DAMON user-space tool for production environments.\nFor example, users can start DAMON to monitor the workload asynchronously, using damo start.\n$ sudo damo start \u0026#34;../masim/masim ../masim/configs/stairs.cfg\u0026#34; Then, users can collect and show the snapshot in various visualization styles, using damo report access, like below.\n$ sudo damo report access heatmap: 00000000000000000000000000000002777777866[...]3333333333333333333333337899965555555447[...]8 # min/max temperatures: -2,210,000,000, 220,010,000, column size: 2.623 MiB 0 addr 85.672 TiB size 20.957 MiB access 0 % age 21.700 s 1 addr 85.672 TiB size 20.664 MiB access 0 % age 22.100 s 2 addr 85.672 TiB size 20.902 MiB access 0 % age 21.800 s 3 addr 85.672 TiB size 20.617 MiB access 0 % age 21.400 s 4 addr 85.672 TiB size 17.754 MiB access 0 % age 1.600 s 5 addr 85.672 TiB size 2.004 MiB access 100 % age 1.100 s 6 addr 85.672 TiB size 4.500 MiB access 0 % age 5.100 s 7 addr 127.047 TiB size 41.785 MiB access 0 % age 12.500 s 8 addr 127.047 TiB size 20.957 MiB access 0 % age 11.900 s 9 addr 127.047 TiB size 4.047 MiB access 0 % age 1.800 s 10 addr 127.047 TiB size 9.516 MiB access 100 % age 2.200 s 11 addr 127.047 TiB size 20.855 MiB access 0 % age 6.500 s 12 addr 127.047 TiB size 5.145 MiB access 0 % age 10 s 13 addr 127.994 TiB size 120.000 KiB access 0 % age 10.900 s 14 addr 127.994 TiB size 8.000 KiB access 35 % age 0 ns 15 addr 127.994 TiB size 4.000 KiB access 0 % age 1.600 s memory bw estimate: 2.250 GiB per second total size: 209.832 MiB monitoring intervals: sample 5 ms, aggr 100 ms $ sudo damo report access --style hot heatmap: 00000000000000000000000000000002666666888[...]3222222222222222888998777777764444444448[...]8 # min/max temperatures: -3,020,000,000, 100,010,000, column size: 2.623 MiB |99999999999999999999999999999999| 9.266 MiB access 100 % 1 s |9999999999999999999999999999| 368.000 KiB access 100 % 200 ms |222222222222222222222222222| 12.000 KiB access 30 % 100 ms |6| 1.273 MiB access 75 % 0 ns |5| 1.035 MiB access 65 % 0 ns |2| 1.730 MiB access 30 % 0 ns |000000000000000000000000000000000| 4.445 MiB access 0 % 1.900 s |0000000000000000000000000000000000| 4.508 MiB access 0 % 2.200 s |00000000000000000000000000000000000| 120.000 KiB access 0 % 4.600 s |000000000000000000000000000000000000| 20.168 MiB access 0 % 5.900 s |000000000000000000000000000000000000| 17.082 MiB access 0 % 7.800 s |00000000000000000000000000000000000000| 20.512 MiB access 0 % 13.100 s |00000000000000000000000000000000000000| 4.398 MiB access 0 % 15.700 s |000000000000000000000000000000000000000| 41.785 MiB access 0 % 20.700 s |000000000000000000000000000000000000000| 20.414 MiB access 0 % 29.600 s |000000000000000000000000000000000000000| 20.957 MiB access 0 % 29.900 s |000000000000000000000000000000000000000| 20.898 MiB access 0 % 29.900 s |0000000000000000000000000000000000000000| 20.871 MiB access 0 % 30.200 s memory bw estimate: 2.300 GiB per second total size: 209.832 MiB monitoring intervals: sample 5 ms, aggr 100 ms $ sudo damo report access --style recency-percentiles # total recency percentiles \u0026lt;percentile\u0026gt; \u0026lt;idle time\u0026gt; 0 0 ns | | 1 0 ns | | 25 12.500 s |***** | 50 24.900 s |********** | 75 48.800 s |******************* | 99 49.100 s |********************| 100 49.100 s |********************| memory bw estimate: 2.317 GiB per second total size: 209.832 MiB monitoring intervals: sample 5 ms, aggr 100 ms Users can also periodically collect snapshots and save those as a file, using damo record. For example, the below command collects the snapshot three times with a five seconds interval between each snapshot, and saves the output as a file named damon.data. The size of the file is much smaller than the entire results record. damo report access can be used for further visualizing the saved data.\n$ sudo damo record --snapshot 5s 3 [...] $ ls -alh ./damon.data -rw------- 1 root root 1.3K Jun 8 12:18 ./damon.data $ sudo damo report access --input_file ./damon.data --style recency-percentiles kdamond 0 / context 0 / scheme 0 / target id None / recorded for 100 ms from 485947 h 18 m 12.956 s # total recency percentiles \u0026lt;percentile\u0026gt; \u0026lt;idle time\u0026gt; 0 0 ns | | 1 0 ns | | 25 16.800 s |*** | 50 22.100 s |**** | 75 1 m 30.300 s |******************* | 99 1 m 30.600 s |********************| 100 1 m 30.600 s |********************| memory bw estimate: 2.212 GiB per second total size: 209.934 MiB monitoring intervals: sample 5 ms, aggr 100 ms kdamond 0 / context 0 / scheme 0 / target id None / recorded for 100 ms from 485947 h 18 m 18.051 s # total recency percentiles \u0026lt;percentile\u0026gt; \u0026lt;idle time\u0026gt; 0 0 ns | | 1 0 ns | | 25 11.600 s |** | 50 24.700 s |***** | 75 1 m 35.300 s |******************* | 99 1 m 35.600 s |********************| 100 1 m 35.600 s |********************| memory bw estimate: 2.176 GiB per second total size: 209.934 MiB monitoring intervals: sample 5 ms, aggr 100 ms kdamond 0 / context 0 / scheme 0 / target id None / recorded for 100 ms from 485947 h 18 m 23.139 s # total recency percentiles \u0026lt;percentile\u0026gt; \u0026lt;idle time\u0026gt; 0 0 ns | | 1 0 ns | | 25 11.800 s |** | 50 23 s |**** | 75 1 m 40.300 s |******************* | 99 1 m 40.600 s |********************| 100 1 m 40.600 s |********************| memory bw estimate: 2.175 GiB per second total size: 209.934 MiB monitoring intervals: sample 5 ms, aggr 100 ms DAMON has gained more features including page level monitoring and intervals auto-tuning. Snapshot collecting and visualization features (damo report access and damo record) also advanced together to support the modern features. With intervals auto-tuning, we believe DAMON can be enabled on production environments always, and snapshot-based data collection and visualization can be useful for system observability.\nMeanwhile, heatmap visualization was not actively updated following the new DAMON features. It was just not working at all for snapshot data. Though snapshot based access visualization was proven to be useful, we recently learned the old style heatmap visualization is what allows people to get the intuitive glance of the access pattern in an easy way. We therefore started working on updating the damo report heatmap to support the modern features, starting from v2.8.3.\nModernized damo report heatmap By v2.8.4, we expect damo report heatmap will show reliable and useful heatmap visualization of snapshot data. It shows the access frequency of each memory region like it was doing before. But if the input is the DAMON monitoring results of snapshot[s] rather than entire results of every moment, it fills the timeline based on the \u0026lsquo;age\u0026rsquo; information of the region on the snapshots. For example, the three snapshots data, which is collected above, can be visualized as a heatmap like below.\n$ sudo damo report heatmap -i damon.data [...] 00000000000000000000000000000000000000000000000000 0000000000000000000 0000000000000000000000000000000000000000000000000000000000 0000000000000000000 00000000000000000000000000000000000000001100000000000000058881000000000000000000 00000000000000000000000000000000000000000000000000000000000000 0000000000000000 00000000000000000000000000000000000000012000000000000000000007887000000000000000 00000000000000000000000000000000000000000000000000000000000000000 000000000000 00000000000000000000000000000000000000000020000000000000000000000888500000000000 # access_frequency: 0123456789 # x-axis: space [85.672 TiB, 85.672 TiB) (202.891 MiB) # y-axis: time [485947 h 16 m 42.456 s, 485947 h 18 m 23.239 s) (1 m 40.783 s) # resolution: 80x40 (2.536 MiB and 2.520 s for each character) The format is the same as the above record-based heatmap. But, now there are blank characters. Those are memory regions that we cannot find the access information from the snapshot. Each snapshot shows the access frquency of each region, and how long the access frequency was kept. Let\u0026rsquo;s say there is an oldest snapshot saying the first 100 MiB region was not accessed at all for 10 seconds, and the next 10 MiB region was accessed with a high access frequency level for 5 seconds. Then, we cannot know what was the access frequency of the 10 MiB region before the 5 seconds. The blank columns are showing that.\nIn more detail, The first three lines of the output are made from the first (oldest) snapshot. The snapshot found about 10 MiB of the region (58881) were accessed frequently, for about 2.5 seconds. The snapshot says the access frequency was kept only for about 2.5 seconds, hence it\u0026rsquo;s access frequency of the past is unknown, so filled with blank columns. This matches with our understanding of masim program\u0026rsquo;s access pattern.\nNext two lines (fourth and fifth) are probably made with the second snapshot. The third snapshot should made the last two lines. The expected masim program\u0026rsquo;s access pattern is continuing to be found there.\nThe latest version of damo report heatmap also supports page level monitoring and intervals auto-tuning. If the snapshot is captured with page level DAMOS filters, users can plot the heatmap for only DAMOS filters-passed pages, by passing --df_passed option to damo report heatmap. The command also understands monitoring intervals auto-tuning feature, and hence it handles the dynamically changed intervals in an appropriate way, when handling the \u0026lsquo;age\u0026rsquo; information.\nThe updated version of the code is available at the next branch, and will be released as v2.8.4 in near future.\nWrapup The classic heatmap visualization was not actively updated since DAMON changed its strategy for monitoring from full access observation records to partial time information-captured snapshots. Now the classic heatmap visualization is modernized to support the snapshots use case, and expected to be useful at understanding overall access patterns at a glance.\n","permalink":"https://damonitor.github.io/posts/damo_heatmap_modernization_2025_06/","summary":"TL; DR: damo report heatmap has recently advanced to support modern DAMON features including age tracking, snapshots, page level monitoring, and monitoring intervals auto-tuning. It will help users intuitively understand the monitored access patterns at a glance.\nDAMON in The Past: Full Recording based Monitoring At the beginning, DAMON was providing only the access frequency of each memory region in real time. Hence heatmap visualization, which shows the access frequency of each memory area in the timeline was the first and one of the best ways to see the access pattern.","title":"`damo report heatmap` modernization for snapshots, page level monitoring and intervals auto-tuning"},{"content":"TL; DR: try --draw_range all option of damo report heatmap if it shows not what you expected. --interactive_edit option can also be helpful, like below.\nProblem: Scoping of Huge Time/Space damo report heatmap outputs sometimes show no expected access pattern. It is sometimes just entirely black, or shows some access pattern but not what the user expected. This post is for explaining the reason and how you can work around.\nThe problem usually happens when user tries to draw the heatmap for access pattern that recorded for a virtual address space of a process. Each virtual address space is very huge. And usually those have two wide regions that not really mapped to physical memory. That\u0026rsquo;s because the stack and the heap are mapped from the two ends of the address space, and do so for mmap()-ed regions from the middle of the space. The resulting mapping looks like below:\n\u0026lt;heap\u0026gt; \u0026lt;BIG UNMAPPED REGION 1\u0026gt; \u0026lt;uppermost mmap()-ed region\u0026gt; (small mmap()-ed regions and munmap()-ed regions) \u0026lt;lowermost mmap()-ed region\u0026gt; \u0026lt;BIG UNMAPPED REGION 2\u0026gt; \u0026lt;stack\u0026gt; If we try to draw the heatmap for the entire address space, the two wide unmapped regions will cover most of the graph. And it will only show black, because the unmapped wide regions will never get accessed.\nSolution: Guide, Manual, Automatic, and Interactive Scope Setup To draw heatmap for only the address ranges that meaningful accesses were made, damo report heatmap allows users to specify what address ranges they want draw heampa for, using --address_range option.\nBecause it is not easy to know to what address range specific data objects of the program is mapped, the command also provides a few features for helping it.\ndamo report record_info shows high level information about the record file. The three address ranges excepting the two huge gaps are included there. For example:\n$ sudo ./damo report record_info --info ./damon.data target_id:0 time: 67900017466000-67959790343999 (59.773 s) region 0: 00000093925306740736-00000093925678731264 (354.758 MiB) region 1: 00000139965707698176-00000139965814927360 (102.262 MiB) region 2: 00000140730699452416-00000140730699587584 (132.000 KiB) To help easier selection, damo report heatmap also provides --draw_range option. As of this writing, the option allows two inputs, namely hottest and all. If all is passed, damo will draw heatmaps for all the mapped regions. If hottest is passed to the option, damo will find a mapped region that most access has happened, and draw the heatmap for only the hottest region.\nAs of this writing, damo report heatmap works as --draw_range hottest is given by default. Note that that the default behavior could be changed in future. Also, --draw_range option will be available from v2.7.2.\nStarting from v2.8.7, damo report heatmap will support interactive --address_range and --time_range modification, when --interactive_edit option is passed. It can also help you explore the huge time/space with the map.\nWrapup So, if damo report heatmap output is different from what you expected, please try running it again with --draw_range all. If you don\u0026rsquo;t want to draw all the maps but hottest one, please use --draw_range hottest. Or, if you want to draw the map for only specific region, use --guide option for high level information that will help you what region you really want to understand in dtail, and use --address_range option.\n","permalink":"https://damonitor.github.io/posts/why_the_heatmap_is_not_showing_the_expected_access_patterns/","summary":"TL; DR: try --draw_range all option of damo report heatmap if it shows not what you expected. --interactive_edit option can also be helpful, like below.\nProblem: Scoping of Huge Time/Space damo report heatmap outputs sometimes show no expected access pattern. It is sometimes just entirely black, or shows some access pattern but not what the user expected. This post is for explaining the reason and how you can work around.","title":"Why the heatmap is not showing the expected access patterns?"},{"content":"We\u0026rsquo;re working on making DAMON to be used for page level properties based access monitoring. The idea is to let users describe specific page level properties that are interested in, and provides the size of the type of memory in each regions that DAMON found unique access pattern.\nHence, users can know how much of memory of specific access temperature is having the type. For example, you can know how much of memory that not accessed for more than 20 minutes are having how much file-backed pages of a cgroup.\nAn RFC patch series for implementing this feature has posted a few days ago, and available at the mailing list. The patches assembled tree is also available at the DAMON development tree. Note that there is a bug that restricts some usage of the features. The fix is available at mailing list.\nDAMON user-space tool, damo is also updated to support the feature and provide a dedicated visualization The updated damo is released as v2.6.1. Refer to the USAGE for detailed usages.\nWith the version of damo on a kernel that built with a kernel tree that based on latest version of DAMON development tree of manually applied the RFC patch series and the fix, therefore, you can try the feature.\nFor example, you can know how much of anonymous and PG_young pages reside in regions of different acces patterns, like below:\n$ sudo ./damo report access --damos_filter anon nomatching --damos_filter young nomatching heatmap: 00000000000000000000000000000000000000000000000000019999999999999622222222222222 # min/max temperatures: -144,860,000,000, -96,980,000,000, column size: 441.765 MiB 0 addr 29.355 GiB size 4.475 GiB access 0 % age 24 m 8.600 s anon and young 0 B 1 addr 33.830 GiB size 5.958 GiB access 0 % age 24 m 7.900 s anon and young 0 B 2 addr 39.788 GiB size 5.972 GiB access 0 % age 24 m 7.300 s anon and young 0 B 3 addr 45.760 GiB size 5.953 GiB access 0 % age 24 m 3.700 s anon and young 24.000 KiB 4 addr 51.714 GiB size 5.978 GiB access 0 % age 16 m 9.800 s anon and young 16.000 KiB 5 addr 57.692 GiB size 5.986 GiB access 0 % age 21 m 50.700 s anon and young 64.000 KiB 6 addr 63.678 GiB size 194.375 MiB access 0 % age 21 m 13.200 s anon and young 0 B total size: 34.513 GiB From the above example output, for example, we can know there are 64 KiB anonymous and young pages, in a memory region of about 6 GiB, that DAMON finds as not accessed for about 22 minutes.\nThe example output is retrieved on a machine that doing nearly nothing, so it looks quite boring. On systems having realistic workloads that dynamic, however, more interesting findings and optimizations based on those would be available.\n","permalink":"https://damonitor.github.io/posts/damon_sz_filter_passed/","summary":"We\u0026rsquo;re working on making DAMON to be used for page level properties based access monitoring. The idea is to let users describe specific page level properties that are interested in, and provides the size of the type of memory in each regions that DAMON found unique access pattern.\nHence, users can know how much of memory of specific access temperature is having the type. For example, you can know how much of memory that not accessed for more than 20 minutes are having how much file-backed pages of a cgroup.","title":"Upcoming feature: Page level peroperties based access monitoring"},{"content":"damo v2.5.7 is released on 2024-11-25. Two new major features on this version are temperature-based regions filtering and formatting.\nTemperature \u0026ldquo;Temperature\u0026rdquo; of each memory region represents relative access hotness of the region. It is calculated as weighted sum of size, access rate (a.k.a nr_accesses) and age of each region. By default, the weights for the three properties are 0, 100, and 100. Users can manually set it using --temperature_weights option.\nTemperature-based regions filtering This feature allows users filter regions of specific temperature range. For example, below shows the access pattern of a Java-based server workload\u0026rsquo;s access pattern in the temperature range to total size of regions of the range histogram visualization format (a.k.a temperature-sz-hist style).\n$ sudo damo report access --style temperature-sz-hist \u0026lt;temperature\u0026gt; \u0026lt;total size\u0026gt; [-571000000000, -509579999000) 108.000 KiB |* | [-509579999000, -448159998000) 4.000 KiB |* | [-448159998000, -386739997000) 20.000 KiB |* | [-386739997000, -325319996000) 0 B | | [-325319996000, -263899995000) 84.000 KiB |* | [-263899995000, -202479994000) 0 B | | [-202479994000, -141059993000) 3.271 GiB |** | [-141059993000, -79639992000) 2.953 GiB |** | [-79639992000, -18219991000) 10.412 GiB |****** | [-18219991000, 43200010000) 40.116 GiB |********************| [43200010000, 104620011000) 84.000 KiB |* | total size: 56.752 GiB From the result, we can know there are warm memory regions of about 40 GiB total size, and hot memory region of 84 KiB total size. Now users would want to know further what access pattern is inside the 40 GiB regions. Users can use the temperature filter feature for the purpose, like below.\n$ sudo damo report access --temperature -18219991000 43200010000 --style temperature-sz-hist \u0026lt;temperature\u0026gt; \u0026lt;total size\u0026gt; [-18,000,000,000, -11,879,999,000) 3.874 GiB |**** | [-11,879,999,000, -5,759,998,000) 11.585 GiB |********** | [-5,759,998,000, 360,003,000) 23.979 GiB |********************| [360,003,000, 6,480,004,000) 693.363 MiB |* | [6,480,004,000, 12,600,005,000) 40.000 KiB |* | [12,600,005,000, 18,720,006,000) 100.000 KiB |* | [18,720,006,000, 24,840,007,000) 0 B | | [24,840,007,000, 30,960,008,000) 8.000 KiB |* | [30,960,008,000, 37,080,009,000) 28.000 KiB |* | [37,080,009,000, 43,200,010,000) 12.000 KiB |* | [43,200,010,000, 49,320,011,000) 84.000 KiB |* | total size: 40.116 GiB Note that damo report access supports more types of region filters. Refer to the usage document for more details.\nRegions formatting with temperature The filtered histogram shows more details. In some cases, users may want to further know not only the total size but every detail of each region, including address, size, access rate, age, and the calculated temperature. The size and address would be specifically interesting if the user is interested in some contiguity-related optimizations like THP.\ndamo report access allows users formatting the output in a flexible way. From damo v2.5.7, the command supports showing temperature of each region. For example, users can show each region\u0026rsquo;s start address, size, access rate, age, and the temperature like below.\n$ sudo damo report access --format_region \u0026#34;\u0026lt;start address\u0026gt; \u0026lt;size\u0026gt; \u0026lt;access rate\u0026gt; \u0026lt;age\u0026gt; \u0026lt;temperature\u0026gt;\u0026#34; 16.219 GiB 93.281 MiB 0 % 22 s -2,200,000,000 16.310 GiB 32.000 KiB 100 % 1 m 36 s 9,600,010,000 16.310 GiB 28.000 KiB 0 % 2 s -,200,000,000 16.310 GiB 32.000 KiB 40 % 4 s 400,004,000 16.310 GiB 84.000 KiB 10 % 0 ns 1,000 [...] ","permalink":"https://damonitor.github.io/posts/damo_2_5_7_features/","summary":"damo v2.5.7 is released on 2024-11-25. Two new major features on this version are temperature-based regions filtering and formatting.\nTemperature \u0026ldquo;Temperature\u0026rdquo; of each memory region represents relative access hotness of the region. It is calculated as weighted sum of size, access rate (a.k.a nr_accesses) and age of each region. By default, the weights for the three properties are 0, 100, and 100. Users can manually set it using --temperature_weights option.","title":"damo v2.5.7 new features: temperature filtering and formatting"},{"content":"The initial version of this post was initially posted to DAMON mailing list as https://lore.kernel.org/20241108232536.73843-1-sj@kernel.org\nPosting it here too, for visibility and after-posting updates for any needs.\nOne of common issues that I received from DAMON users is that DAMON\u0026rsquo;s monitoring results show hot regions much less than expected. Specifically, the users find regions of only zero or low \u0026rsquo;nr_accesses\u0026rsquo; value from the DAMON-generated access pattern snapshots.\nIn some cases, it turned out the problem can be alleviated by tuning DAMON parameters or changing the way to interpret the results. I\u0026rsquo;d like to share the details and possible future improvements here.\nNote that I\u0026rsquo;m not saying this is users\u0026rsquo; tuning fault. I admit that the real root cause of the issue is the poor interface and lack of guides that makes correct tuning difficult, and the suboptimality of DAMON\u0026rsquo;s mechanisms. We will continue working on advancing it in long term. Sharing some of the plans and status at the end of this email.\nTL; DR Users show only low or zero nr_accesses regions mainly because they set \u0026lsquo;aggregation intrval\u0026rsquo; too short compared to the workload\u0026rsquo;s memory access intensiveness. Please increase the aggregation interval, or treat \u0026rsquo;nr_accesses\u0026rsquo; zero regions of short \u0026lsquo;age\u0026rsquo; as hot regions.\nNow let\u0026rsquo;s walk through more details. The below sections assume you\u0026rsquo;re familiar with DAMON\u0026rsquo;s monitoring mechanisms including \u0026lsquo;Access Frequency Monitoring\u0026rsquo;, \u0026lsquo;Regions Based Sampling\u0026rsquo;, \u0026lsquo;Adaptive Regions Adjustment\u0026rsquo;, and \u0026lsquo;Age Tracking\u0026rsquo;. You should particularly be familiar with terms including \u0026lsquo;sampling interval\u0026rsquo;, \u0026lsquo;aggregation interval\u0026rsquo;, \u0026rsquo;nr_accesses\u0026rsquo;, and \u0026lsquo;age\u0026rsquo;. If you\u0026rsquo;re not familiar with those, you can refer to the document (https://www.kernel.org/doc/html/next/mm/damon/design.html#monitoring).\nProblem Some users reported that they expected DAMON will be useful for finding both cold memory regions and hot memory regions. And they found it indeed works for finding cold memory regions. But, they found difficulties at finding hot memory regions. Some detailed reported cases are as below.\nCase 1: Proactive Reclamation\nA user tried to use DAMOS for proactively reclaiming cold memory regions. They hence specified maximum access rate (nr_accesses) and minimum age of target regions for the DAMOS scheme as zero and 2 minutes, respectively. It means asking DAMON to find regions that not accessed for two minutes or more and reclaim those as soon as found. If they use DAMON user-space tool, damo, they would used a command like below.\n# damo start --damos_action pageout --damos_access rate 0% 0% --damos_age 2m max The result was as the user expected. The user reported me that they were able to save memory without performance degradation.\nMy test setup also showed good results from similar DAMOS schemes, even though my setup was using even more aggressive approach (minimum age as 5 seconds).\nCase 2: Heatmap Visualization\nA user has recorded data access pattern of their workloads using DAMON user-space tool, damo, and draw heatmap using \u0026lsquo;damo report heatmap\u0026rsquo; command. The workload was active and therefore the user expected the heatmap to show some heats. But the output was showing nearly zero heats. It was nearly always only dark.\nOn my test setup, some workloads indeed showed very dark heatmap. But on multiple workloads, meaningful access patterns were identifiable using the heatmaps (https://damonitor.github.io/test/result/visual/next/rec.heatmap.1.png.html).\nCase 3: Prioritizing Hot Pages\nA user tried to use DAMOS for finding hot memory regions and make a prioritization action like backing the regions with THP, migrating the pages from CXL node to DRAM node, or moving the page from inactive LRU list to active LRU list. They hence specified minimum access rate (nr_accesses) for the DAMOS scheme with a high value. For example, a command like below was used:\n# damo start --damos_action hugepage --damos_access_rate 50% max $(pidof workload) However, they found DAMOS is doing nearly nothing. They reduced the minimum access_rate, and situation has been better, but still DAMOS was not finding expected amount of hot memory regions.\nCase 2 and 3 mean that they expected to show regions of high \u0026rsquo;nr_accesses\u0026rsquo; value from DAMON-generated access pattern snapshots. But it showed only zero or low \u0026rsquo;nr_accesses\u0026rsquo; values.\nOn my test setup (https://damonitor.github.io/posts/damon_evaluation/), similar approach showed good results, though.\nOne common thing that I found from the reports was that they were using default values for \u0026lsquo;aggregation interval\u0026rsquo;, which is 100 milliseconds. My test setup is also using the default values.\nMeaning of \u0026lsquo;aggregation interval\u0026rsquo; For every \u0026lsquo;aggregation interval\u0026rsquo;, DAMON generates one complete access pattern snapshot. That means \u0026lsquo;aggregation interval\u0026rsquo; can be though of as the time amount of information that a single snapshot can contain. The value should hence be decided depending on how much information the user wants each snapshot to have. If it is too short, each snapshot contains nearly no information, so not be useful. If it is too long, each snapshot contains too much information that coarsely cumulated, so again not useful.\nThe meaningful length of time depends on the data access intensiveness of the workload. The intensiveness should be measured with two factors: frequency and footprint.\nThe frequency is how frequently the workload is making accesses. If the workload makes zero or only a low number of accesses per given \u0026lsquo;aggregation interval\u0026rsquo;, the snapshot will of course show zero or low number of accesses, due to \u0026lsquo;Access Frequency Monitoring\u0026rsquo; mechanism.\nThe footprint is the total amount of memory that the workload has accessed at least once. Due to \u0026lsquo;Region Based Sampling\u0026rsquo; mechanism, it should be meaningfully large compared to the size of total monitoring target regions. For example, if DAMON is requested to monitor 1 TiB memory and the workload is accessing 1 MiB region of it, DAMON\u0026rsquo;s sampling based approach will have difficulty at finding the 1 MiB region. \u0026lsquo;Adaptive Regions Adjustment\u0026rsquo; mechanism will make DAMON to find the 1 MiB region eventually. But it will take time. The workload could stop accessing the 1 MiB region and starts accessing another 1 MiB region before the \u0026lsquo;Adaptive Regions Adjustment\u0026rsquo; mechanism finds it.\nNote that the frequency and footprint of accesses from workloads would depend on not only the source code but also many factors. The system\u0026rsquo;s total memory bandwidth, the extent of load and noisy neighbor workloads could be a few examples.\nSo, if the desired maximum \u0026rsquo;nr_accesses\u0026rsquo; on each snapshot is fixed, \u0026lsquo;aggregation interval\u0026rsquo; should be increased as the access aggressiveness is decreased, and vice versa.\nRoot Causes 1: Suboptimal Aggregation Interval Tuning I suspect in many of the reported problematic cases, the \u0026lsquo;aggregation interval\u0026rsquo; was too short. The default value is good enough for my test setup, but it would need tuning on different systems. Especially on large systems having limited bandwidth, 100 millisecond \u0026lsquo;aggregation interval\u0026rsquo; could be not long enough to make meaningful amount of accesses in terms of both frequency and footprint within the interval.\nI actually suggested some of the reporters to increase \u0026lsquo;aggregation interval\u0026rsquo;, and it was helpful at alleviating the issue in some degree. I cannot know exactly who are using DAMON with what parameters. But at least one open-sourced usage from HMSDK is setting \u0026lsquo;aggregation interval\u0026rsquo; as two seconds (https://github.com/skhynix/hmsdk/blob/hmsdk-v3.0/tools/gen_migpol.py#L14).\nRoot Cause 2: Ignorance of Recency (\u0026lsquo;age\u0026rsquo;) The \u0026rsquo;nr_accesses\u0026rsquo; represents the access frequency, and hence it is natural to assume hot memory regions would have high \u0026rsquo;nr_accesses\u0026rsquo;.\nDAMON provides not only frequency. It also informs users how long the \u0026rsquo;nr_accesses\u0026rsquo; of the regions is maintained, namely \u0026lsquo;age\u0026rsquo;, using \u0026lsquo;Age Tracking\u0026rsquo; mechanism. If \u0026rsquo;nr_accesses\u0026rsquo; is non-zero, \u0026lsquo;age\u0026rsquo; can be used to calculate actual access hotness of the region (if we turn fire on for longer time, the temperature will be higher). If \u0026rsquo;nr_accesses\u0026rsquo; is zero, \u0026lsquo;age\u0026rsquo; can represent a sort of the recency information (when the regions has last accessed).\nThe recency information can be useful for finding cold pages like case 1 (proactive reclaim). The opposite of finding cold pages is finding hot pages, so the recency information can also be used for finding relatively hot pages. In other words, if for any reason DAMON cannot generate a snapshot having enough non-zero \u0026rsquo;nr_accesses\u0026rsquo; divergence for given purpose, users could further differentiate hot regions among zero \u0026rsquo;nr_accesses\u0026rsquo; regions, using \u0026lsquo;age\u0026rsquo; information. It would be not ideal, but still reasonable like a sort of LRU-based appraoches could be.\nSo I suspect some of the problems occur from not using \u0026lsquo;age\u0026rsquo; for zero \u0026rsquo;nr_accesses\u0026rsquo; regions. Actually reports from case 3 (prioritizing hot pages), which were successful only on my test setup, were commonly using non-zero minimum \u0026rsquo;nr_accesses\u0026rsquo; for DAMOS schemes, so were ignoring \u0026lsquo;age\u0026rsquo; of zero \u0026rsquo;nr_accesses\u0026rsquo; regions. Meanwhile, case 2 (proactive reclaim) was using \u0026lsquo;age\u0026rsquo; information for zero \u0026rsquo;nr_accesses\u0026rsquo; regions, and no negative results have reported so far.\nThis might seem like not addressing case 2 (heatmap visualization), because heatmap shows \u0026rsquo;nr_accesses\u0026rsquo; change of regions over time. But if the record is collected for long time, regions that shown non-zero \u0026rsquo;nr_accesses\u0026rsquo; for a short period may look a very small dot on the low-resolution picture that not easy to shown with human eyes. The users might be able to get different results using \u0026lsquo;age\u0026rsquo; information on the collected snapshot, like recency or temperation based histogram (https://github.com/damonitor/damo/blob/v2.5.4/USAGE.md#access-report-styles).\nTuning Guide Based on above root cause theories, I suggest to try below tuning guides.\nIf you show DAMON is not working well at finding hot pages,\nEnsure your workload is making meaningfully intensive data accesses. Gradually increase aggregation interval and show if it makes change. Try using \u0026lsquo;age\u0026rsquo; information even if \u0026rsquo;nr_accesses\u0026rsquo; is zero. If nothing works, report the problem to sj@kernel.org, damon@lists.linux.dev and/or linux-mm@kvack.org. If increasing aggregation interval alleviates your problem, you can further consider increasing \u0026lsquo;sampling interval\u0026rsquo;. If it doesn\u0026rsquo;t harm the quality of the access pattern snapshots, having low \u0026lsquo;sampling interval\u0026rsquo; will only increase DAMON\u0026rsquo;s CPU usage.\nFor using \u0026lsquo;age\u0026rsquo; information of zero \u0026rsquo;nr_accesses\u0026rsquo; regions, different approaches could be used for profiling use case and DAMOS use case. For profiling use case, users can try reading recency or access temperature based histograms (https://github.com/damonitor/damo/blob/v2.5.4/USAGE.md#access-report-styles) of snapshots from record, or live-captured snapshots.\nIf the use case is for DAMOS, applying the \u0026lsquo;age\u0026rsquo; information on DAMOS target access pattern would be straightforward. Using DAMOS Quotas together can be useful, since it provides its own under-quota-prioritization logic that utilizes \u0026lsquo;age\u0026rsquo; information for zero \u0026rsquo;nr_accesses\u0026rsquo; regions, and further provides auto-tuning of the quota for given target metric/value.\nTuning Example The initial version of this section is also posted to the mailing list: https://lore.kernel.org/20241202175459.2005526-1-sj@kernel.org\nI tried monitoring the access patterns on the physical address space of a system running a real-world server workload. I was able to reproduce the reported poor quality of hot pages detection using default parameter. And I was also able to improve the quality following the above tuning guide. I\u0026rsquo;m sharing the details as an example.\n5ms/100ms intervals: Reproduce Problem Initially, I captured the access pattern snapshot on the physical address space using DAMON, with the default interval parameters (5 milliseconds and 100 milliseconds for the sampling and the aggregation intervals, respectively). I wait ten minutes after starting DAMON, to show a meaningful time-wise access patterns.\n# damo start # sleep 600 # damo record --snapshot 0 1 # damo stop Then, I listed the DAMON-found regions of different access patterns, sorted by the access temperature. Access temperature is calculated as a weighted sum of the access frequency and the age of the region. If the access frequency is 0 %, the temperature is multipled by minus one. That is, if a region is not accessed, it gets minus temperature and it gets lower as not accessed for longer time. The sorting is in temperature-ascendint order, so the region at the top of the list is the coldest, and the one at the bottom is the hottest one.\n# damo report access --sort_regions_by temperature 0 addr 16.052 GiB size 5.985 GiB access 0 % age 5.900 s # coldest 1 addr 22.037 GiB size 6.029 GiB access 0 % age 5.300 s 2 addr 28.065 GiB size 6.045 GiB access 0 % age 5.200 s 3 addr 10.069 GiB size 5.983 GiB access 0 % age 4.500 s 4 addr 4.000 GiB size 6.069 GiB access 0 % age 4.400 s 5 addr 62.008 GiB size 3.992 GiB access 0 % age 3.700 s 6 addr 56.795 GiB size 5.213 GiB access 0 % age 3.300 s 7 addr 39.393 GiB size 6.096 GiB access 0 % age 2.800 s 8 addr 50.782 GiB size 6.012 GiB access 0 % age 2.800 s 9 addr 34.111 GiB size 5.282 GiB access 0 % age 2.300 s 10 addr 45.489 GiB size 5.293 GiB access 0 % age 1.800 s # hottest total size: 62.000 GiB The list shows not seemingly hot regions, and only minimum access pattern diversity. Every region has zero access frequency. The number of region is 10, which is the default min_nr_regions value. Size of each region is also nearly idential. We can suspect this is because \u0026ldquo;adaptive regions adjustment\u0026rdquo; mechanism was not well working. As the guide suggested, we can get relative hotness of regions using \u0026lsquo;age\u0026rsquo; as the recency information. That would be better than nothing, but given the fact that the longest age is only about 6 seconds while we waited about ten minuts, it is unclear how useful this will be.\nThe temperature ranges to total size of regions of each range histogram visualization of the results also shows no interesting distribution pattern.\n# damo report access --style temperature-sz-hist \u0026lt;temperature\u0026gt; \u0026lt;total size\u0026gt; [-,590,000,000, -,549,000,000) 5.985 GiB |********** | [-,549,000,000, -,508,000,000) 12.074 GiB |********************| [-,508,000,000, -,467,000,000) 0 B | | [-,467,000,000, -,426,000,000) 12.052 GiB |********************| [-,426,000,000, -,385,000,000) 0 B | | [-,385,000,000, -,344,000,000) 3.992 GiB |******* | [-,344,000,000, -,303,000,000) 5.213 GiB |********* | [-,303,000,000, -,262,000,000) 12.109 GiB |********************| [-,262,000,000, -,221,000,000) 5.282 GiB |********* | [-,221,000,000, -,180,000,000) 0 B | | [-,180,000,000, -,139,000,000) 5.293 GiB |********* | total size: 62.000 GiB In short, the result is very similar to the reported problems: poor quality monitoring results for hot regions detection. According to the above guide, this is due to the too short aggregation interval.\n100ms/2s intervals: Starts Showing Small Hot Regions Following the guide, I increased the interval 20 times (100 milliseocnds and 2 seconds for sampling and aggregation intervals, respectively).\n# damo start -s 100ms -a 2s # sleep 600 # damo record --snapshot 0 1 # damo stop # damo report access --sort_regions_by temperature 0 addr 10.180 GiB size 6.117 GiB access 0 % age 7 m 8 s # coldest 1 addr 49.275 GiB size 6.195 GiB access 0 % age 6 m 14 s 2 addr 62.421 GiB size 3.579 GiB access 0 % age 6 m 4 s 3 addr 40.154 GiB size 6.127 GiB access 0 % age 5 m 40 s 4 addr 16.296 GiB size 6.182 GiB access 0 % age 5 m 32 s 5 addr 34.254 GiB size 5.899 GiB access 0 % age 5 m 24 s 6 addr 46.281 GiB size 2.995 GiB access 0 % age 5 m 20 s 7 addr 28.420 GiB size 5.835 GiB access 0 % age 5 m 6 s 8 addr 4.000 GiB size 6.180 GiB access 0 % age 4 m 16 s 9 addr 22.478 GiB size 5.942 GiB access 0 % age 3 m 58 s 10 addr 55.470 GiB size 915.645 MiB access 0 % age 3 m 6 s 11 addr 56.364 GiB size 6.056 GiB access 0 % age 2 m 8 s 12 addr 56.364 GiB size 4.000 KiB access 95 % age 16 s 13 addr 49.275 GiB size 4.000 KiB access 100 % age 8 m 24 s # hottest total size: 62.000 GiB # damo report access --style temperature-sz-hist \u0026lt;temperature\u0026gt; \u0026lt;total size\u0026gt; [-42,800,000,000, -33,479,999,000) 22.018 GiB |***************** | [-33,479,999,000, -24,159,998,000) 27.090 GiB |********************| [-24,159,998,000, -14,839,997,000) 6.836 GiB |****** | [-14,839,997,000, -5,519,996,000) 6.056 GiB |***** | [-5,519,996,000, 3,800,005,000) 4.000 KiB |* | [3,800,005,000, 13,120,006,000) 0 B | | [13,120,006,000, 22,440,007,000) 0 B | | [22,440,007,000, 31,760,008,000) 0 B | | [31,760,008,000, 41,080,009,000) 0 B | | [41,080,009,000, 50,400,010,000) 0 B | | [50,400,010,000, 59,720,011,000) 4.000 KiB |* | total size: 62.000 GiB DAMON found two distinct 4 KiB regions that pretty hot. The regions are also well aged. The hottest 4 KiB region was keeping the access frequency for about 8 minutes, and the coldest region was keeping no access for about 7 minutes. The distribution on the histogram also looks like having a pattern.\nEspecially, the finding of the 4 KiB regions shows DAMON\u0026rsquo;s adaptive regions adjustment is working as designed.\nStill the number of regions is close to the min_nr_regions, and sizes of cold regions are similar, though. Apparently it is improved, but it still has rooms to improve.\n400ms/8s intervals: Pretty Improved Results I further increased the intervals four times (400 milliseconds and 8 seconds for sampling and aggregation intervals, respectively).\n# damo start -s 400ms -a 8s # sleep 600 # damo record --snapshot 0 1 # damo stop # damo report access --sort_regions_by temperature 0 addr 64.492 GiB size 1.508 GiB access 0 % age 6 m 48 s # coldest 1 addr 21.749 GiB size 5.674 GiB access 0 % age 6 m 8 s 2 addr 27.422 GiB size 5.801 GiB access 0 % age 6 m 3 addr 49.431 GiB size 8.675 GiB access 0 % age 5 m 28 s 4 addr 33.223 GiB size 5.645 GiB access 0 % age 5 m 12 s 5 addr 58.321 GiB size 6.170 GiB access 0 % age 5 m 4 s [...] 25 addr 6.615 GiB size 297.531 MiB access 15 % age 0 ns 26 addr 9.513 GiB size 12.000 KiB access 20 % age 0 ns 27 addr 9.511 GiB size 108.000 KiB access 25 % age 0 ns 28 addr 9.513 GiB size 20.000 KiB access 25 % age 0 ns 29 addr 9.511 GiB size 12.000 KiB access 30 % age 0 ns 30 addr 9.520 GiB size 4.000 KiB access 40 % age 0 ns [...] 41 addr 9.520 GiB size 4.000 KiB access 80 % age 56 s 42 addr 9.511 GiB size 12.000 KiB access 100 % age 6 m 16 s 43 addr 58.321 GiB size 4.000 KiB access 100 % age 6 m 24 s 44 addr 9.512 GiB size 4.000 KiB access 100 % age 6 m 48 s 45 addr 58.106 GiB size 4.000 KiB access 100 % age 6 m 48 s # hottest total size: 62.000 GiB # damo report access --style temperature-sz-hist \u0026lt;temperature\u0026gt; \u0026lt;total size\u0026gt; [-40,800,000,000, -32,639,999,000) 21.657 GiB |********************| [-32,639,999,000, -24,479,998,000) 17.938 GiB |***************** | [-24,479,998,000, -16,319,997,000) 16.885 GiB |**************** | [-16,319,997,000, -8,159,996,000) 586.879 MiB |* | [-8,159,996,000, 5,000) 4.946 GiB |***** | [5,000, 8,160,006,000) 260.000 KiB |* | [8,160,006,000, 16,320,007,000) 0 B | | [16,320,007,000, 24,480,008,000) 0 B | | [24,480,008,000, 32,640,009,000) 0 B | | [32,640,009,000, 40,800,010,000) 16.000 KiB |* | [40,800,010,000, 48,960,011,000) 8.000 KiB |* | total size: 62.000 GiB The number of regions having different access patterns has significantly increased. Size of each region is also more varied. Total size of non-zero access frequency regions is also significantly increased. Maybe this is already good enough to make some meaningful memory management efficieny changes.\n800ms/16s intervals: Another bias Further doubling the intervals (800 milliseconds and 16 seconds for sampling and aggregation intervals, respectively) mor improved the hot regions detection, but starts looking degrading cold regions detection.\n# damo start -s 800ms -a 16s # sleep 600 # damo record --snapshot 0 1 # damo stop # damo report access --sort_regions_by temperature 0 addr 64.781 GiB size 1.219 GiB access 0 % age 4 m 48 s 1 addr 24.505 GiB size 2.475 GiB access 0 % age 4 m 16 s 2 addr 26.980 GiB size 504.273 MiB access 0 % age 4 m 3 addr 29.443 GiB size 2.462 GiB access 0 % age 4 m 4 addr 37.264 GiB size 5.645 GiB access 0 % age 4 m 5 addr 31.905 GiB size 5.359 GiB access 0 % age 3 m 44 s [...] 20 addr 8.711 GiB size 40.000 KiB access 5 % age 2 m 40 s 21 addr 27.473 GiB size 1.970 GiB access 5 % age 4 m 22 addr 48.185 GiB size 4.625 GiB access 5 % age 4 m 23 addr 47.304 GiB size 902.117 MiB access 10 % age 4 m 24 addr 8.711 GiB size 4.000 KiB access 100 % age 4 m 25 addr 20.793 GiB size 3.713 GiB access 5 % age 4 m 16 s 26 addr 8.773 GiB size 4.000 KiB access 100 % age 4 m 16 s total size: 62.000 GiB # damo report access --style temperature-sz-hist \u0026lt;temperature\u0026gt; \u0026lt;total size\u0026gt; [-28,800,000,000, -23,359,999,000) 12.294 GiB |***************** | [-23,359,999,000, -17,919,998,000) 9.753 GiB |************* | [-17,919,998,000, -12,479,997,000) 15.131 GiB |********************| [-12,479,997,000, -7,039,996,000) 0 B | | [-7,039,996,000, -1,599,995,000) 7.506 GiB |********** | [-1,599,995,000, 3,840,006,000) 6.127 GiB |********* | [3,840,006,000, 9,280,007,000) 0 B | | [9,280,007,000, 14,720,008,000) 136.000 KiB |* | [14,720,008,000, 20,160,009,000) 40.000 KiB |* | [20,160,009,000, 25,600,010,000) 11.188 GiB |*************** | [25,600,010,000, 31,040,011,000) 4.000 KiB |* | total size: 62.000 GiB It found more non-zero access frequency regions. The number of regions is still much higher than the min_nr_regions, but it is reduced from that of the previous setup. And apparently the distribution seems bit biased to hot regions.\nConclusion Because the workload is live, the above results are not always consistent. But, the tendency of the quality for the interval changes was consistent.\nWith the above experimental tuning results, I conclude the theory and the guide makes sense to at least this workload, and could be applied to similar cases.\nThis also gives us an idea for automated tuning of the intervals. If the interval is too short, results are biased to cold regions. If the interval is too long, results are biased to hot regions. Maybe DAMON can moitor to which direction the current snapshot is biased, and adjust the intervals. I will develop the idea more.\nFuture Plans Again, I admit that the real root cause of the issue is the poor interface and lack of guides that makes correct tuning difficult, and the suboptimality of DAMON\u0026rsquo;s mechanisms. We will continue working on advancing it in long term.\nFor easy use of \u0026lsquo;age\u0026rsquo; information for zero \u0026rsquo;nr_accesses\u0026rsquo; regions, DAMON is already providing its Quotas feature with auto-tuning. We will continue making the auto-tuning more advanced, and adding new features for ease of uses.\nOne obvious hidden root cause is the absence of the guideline. I will collect more inputs and write an official document for this. I actually thought about writing the official document first, but this writing this mail took already over two weeks. So posting this mail first.\nFor interval setting, we are planning a sort of auto-tuning. Like DAMOS quota auto-tuning, users will be able to set more intuitive knobs (e.g., how much diversity of regions they want) instead of directly setting the intervals. Then, DAMON will automatically tune the intervals and other low level knobs. This is in very early stage, so no specific design is made so far, and will take long time. Don\u0026rsquo;t expect delivery of this in near future. Use the tuning guide for short term and/or ask prioritization of this project if needed.\nIn my humble opinion, \u0026lsquo;Adaptive Regions Adjustment\u0026rsquo; mechanism is not a root cause of the issue. Nonetheless, it also has many rooms for improvement that can make DAMON more lightweight and accurate. And any DAMON accuracy improvements would alleviate the hot page detection issue. We plan to do this together, and this is again a long term project that has no specific design yet. Nonetheless, we recently shared two simple short term features for this (https://lore.kernel.org/20241027204910.155254-1-sj@kernel.org). If you are interested in implementing the short term features, please step up.\n","permalink":"https://damonitor.github.io/posts/damon_tuning_guide_for_hot_pages/","summary":"The initial version of this post was initially posted to DAMON mailing list as https://lore.kernel.org/20241108232536.73843-1-sj@kernel.org\nPosting it here too, for visibility and after-posting updates for any needs.\nOne of common issues that I received from DAMON users is that DAMON\u0026rsquo;s monitoring results show hot regions much less than expected. Specifically, the users find regions of only zero or low \u0026rsquo;nr_accesses\u0026rsquo; value from the DAMON-generated access pattern snapshots.\nIn some cases, it turned out the problem can be alleviated by tuning DAMON parameters or changing the way to interpret the results.","title":"A guide to DAMON tuning and results interpretation for hot pages"},{"content":"Starting from Linux v6.9, DAMON provides DAMOS quota auto-tuning. It allows users to set a target metric and value. Then, DAMOS will adjust its aggressiveness (effective quota) to achieve the target.\ndamo users can also use the feature using --damos_quota_goal option. But apparently the usage is not well documented. Maybe it should be documented somewhere on USAGE.md of damo, but I cannot find a good splot for now. So I\u0026rsquo;m explaining the usage in more informal way on this post. Hopefully this will help users while I make more formal writeup on the USAGE.md file.\nFirst, please read the DAMON design document for the feature. Below sections will assume you know the concepts.\nDAMOS self-feedback-driven auto-tuning The usage for self-feedback driven auto-tuning is simple. You can just set the target metric and the value. Currently, only some_mem_psi_us metric is supported for this purpose. For example, below command asks DAMON to page out pages of workload, aiming 500us of some memory pressure stall time per second.\n# damo start $(pidof workload) --damos_action pageout \\ --damos_quota_interval 1s \\ --damos_quota_goal some_mem_psi_us 500us If the memory pressure stall rate of the system is lower than the target (500 microsecond per second), DAMOS will pageout more pages (coldest one first) per second. If the memory pressure stall rate is higher than the target, DAMOS will pageout less pages (coldest one first) per second.\nOnline-updating target Depending on the cases, you might realize the 500us per second memory pressure was not the right target you want. You can update it while DAMON is running online, using damo tune command. For example,\n# damo tune $(pidof workload) --damos_action pageout \\ --damos_quota_interval 1s \\ --damos_quota_goal some_mem_psi_us 750us Reading auto-tuning resulting aggressiveness You may want to know so how aggressively DAMOS is working under the target. For this purpose, you can use effective_bytes DAMON sysfs file. You can read the file via damo report damon, from effective_sz_bytes field. It must be intuitive when you use json format. For example,\n$ sudo damo report damon --json [...] \u0026quot;schemes\u0026quot;: [ [...] \u0026quot;quotas\u0026quot;: { [...] \u0026quot;reset_interval_ms\u0026quot;: \u0026quot;1 s\u0026quot;, [...] \u0026quot;effective_sz_bytes\u0026quot;: \u0026quot;256 B\u0026quot;, [...] On the above example output, DAMON has set the effective size quota as 256 Bytes per second, as a result of the auto-tuning.\nIf you use the human-friendly output, the value is on the first line for quotas. For example,\n$ sudo damo report damon kdamond 0 [...] quotas 0 ns / 0 B / 8 B per 1 s goal 0: metric some_mem_psi_us target 500 current 0 priority: sz 0.1 %, nr_accesses 0.1 %, age 0.1 % [...] The four numbers on the line after quotas line show time quota, size quota, the effective size quota, and quotas reset interval. So in this example, DAMON has set the effective size quota as 8 Bytes per second.\nUser feedback-driven auto-tuning The usage for user feedback driven auto-tuning is similar to system self-feedback. Major difference is that the user should provide the feedback whenever the feedback score is changed. You can use damo tune.\nFor example, let\u0026rsquo;s say your workload writes its performance score (e.g., number of processed queries per second) to score file per second. And you want to do proactive reclaim of the workload, aiming 10,000 of the score.\nYou may first start DAMON.\n# damo start $(pidof workload) --damos_action pageout \\ --damos_quota_interval 1s \\ --damos_quota_goal user_input 10000 $(cat score) Unlike self-feedback case, --damos_quota_goal option receives three arguments when the metric argument is user_input. The additional argument is the current value of the user metric. In this example, we just provide the current content of score file.\nNow, DAMON will start proactive reclamation. You should periodically read the real score and give update to DAMON when needed. This could be done like below.\nwhile true: do now_score=$(cat score) if [ \u0026quot;$last_score\u0026quot; -eq 0 ] || [ \u0026quot;$now_score\u0026quot; -ne \u0026quot;$last_score\u0026quot; ] then # damo tune $(pidof workload) --damos_action pageout \\ --damos_quota_interval 1s \\ --damos_quota_goal user_input 10000 $(cat score) last_score=now_score fi sleep 1 done The above example shell script will read the workload\u0026rsquo;s score every second, and update the feed score for DAMON if some changes are made. If the feed score is same, you don\u0026rsquo;t need to give update to DAMON.\n","permalink":"https://damonitor.github.io/posts/damo_autotune_example/","summary":"Starting from Linux v6.9, DAMON provides DAMOS quota auto-tuning. It allows users to set a target metric and value. Then, DAMOS will adjust its aggressiveness (effective quota) to achieve the target.\ndamo users can also use the feature using --damos_quota_goal option. But apparently the usage is not well documented. Maybe it should be documented somewhere on USAGE.md of damo, but I cannot find a good splot for now. So I\u0026rsquo;m explaining the usage in more informal way on this post.","title":"Auto-tuning DAMOS using `damo`"},{"content":"I just made a DAMON logo using DAMON, like below.\n$ git clone https://github.com/sjp38/masim \u0026amp;\u0026amp; cd masim $ cat damon_pixel_2 11111111 11 11 111111 11111111 11 11 11111111 11111111 1111 11111111 11111111 11 11 11111111 11111111 1111 11111111 $ ./pixels_to_access_config.py ./damon_pixel_2 $((100*1024*1024)) 300 damon.cfg $ sudo damo record \u0026#34;./masim ./configs/stairs.cfg\u0026#34; $ sudo damo report heatmap --output damon.png The output is below:\nThe cropped one:\n","permalink":"https://damonitor.github.io/posts/damon_heatmap_logo/","summary":"I just made a DAMON logo using DAMON, like below.\n$ git clone https://github.com/sjp38/masim \u0026amp;\u0026amp; cd masim $ cat damon_pixel_2 11111111 11 11 111111 11111111 11 11 11111111 11111111 1111 11111111 11111111 11 11 11111111 11111111 1111 11111111 $ ./pixels_to_access_config.py ./damon_pixel_2 $((100*1024*1024)) 300 damon.cfg $ sudo damo record \u0026#34;./masim ./configs/stairs.cfg\u0026#34; $ sudo damo report heatmap --output damon.png The output is below:\nThe cropped one:","title":"Creating DAMON logo using DAMON"},{"content":"This post provides lists of publications and presentations that cover DAMON.\nFeatured Publications and Talks There are quite amount of publications and talks. Below are featured ones for people who unsure what among the resources to use.\nAcademic papers For people who more familiar to academic papers, DAMON papers for Middleware'19 Industry and HPDC'22 are recommended to read and/or cite. The paper for Middleware'19 covers DAMON\u0026rsquo;s monitoring mechanisms and access pattern profiling-guided optimizations. The paper for HPDC'22 extends the coverage to DAMOS (automated access-aware system operations) and user-space driven auto-tuning.\nTalks for beginners and users If you are looking for a resources to start with, below talks are recommended.\nSeongJae Park, Page-level and Fleet-wide Data Access Monitoring for Meta. In Linux Plumbers Refereed Track, Dec 2025. Slides, Video, Link SeongJae Park, Overcoming Observer Effects in Memory Management with DAMON. In Kernel Recipes, Sep 2025. Slides, Video, Link SeongJae Park, DAMON: Kernel Subsystem for Data Access Monitoring and Access-aware System Operations. In Fosdem, Feb 2025. Slides, Video, Link Talks for experts and developers If you want to track recent DAMON developmeent status and plans, below talks are recommended.\nSeongJae Park, DAMON-based Pages Migration for {C,G,X}PU [un]attached NUMA nodes. In Device and Specific PurposeMemory MC at Linux Plumbers, Dec 2025. Slides, Video, Link SeongJae Park, DAMON Requirements for Access-aware MM of Future. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, Mar 2025. Slides, Link, LWN article SeongJae Park, DAMON Updates and Plans: Monitoring Parameters Auot-tuning and Memory Tiering. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, Mar 2025. Slides, Link, LWN article Full list of upcoming and past publications and talks Below is a list of past and upcoming publications and talks covering DAMON. The list is not exhaustive and is compiled to the best of our ability, as some publications or presentations may have been made without the knowledge of the DAMON maintainers. If you find a publication or announcement that should be added to this list, please let us know at sj@kernel.org and/or damon@lists.linux.dev.\n2025\nSeongJae Park, Page-level and Fleet-wide Data Access Monitoring for Meta. In Linux Plumbers Refereed Track, Dec 2025. Slides, Video, Link SeongJae Park, Actionable Data Access Monitoring Output Data and Format. In Linux System Monitoring and Observability MC at Linux Plumbers, Dec 2025. Slides, Video, Link SeongJae Park, DAMON-based Pages Migration for {C,G,X}PU [un]attached NUMA nodes. In Device and Specific PurposeMemory MC at Linux Plumbers, Dec 2025. Slides, Video, Link SeongJae Park, Overcoming Observer Effects in Memory Management with DAMON. In Kernel Recipes, Sep 2025. Slides, Video, Link Asier Gutierrez, Hybrid THP Mechanism. Selective Use of Huge Pages by Hot Applications. In Open Source Summit Europe, Aug 2025. Video, Link SeongJae Park, Self-Driving DAMON/S: Controlled and Automated Access-aware Efficient Systems. In Open Source Summit North America, Jun 2025. Slides, Video, Link SeongJae Park, DAMON Requirements for Access-aware MM of Future. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, Mar 2025. Slides, Link SeongJae Park, DAMON Updates and Plans: Monitoring Parameters Auot-tuning and Memory Tiering. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, Mar 2025. Slides, Link, LWN article SeongJae Park, DAMON: Kernel Subsystem for Data Access Monitoring and Access-aware System Operations. In Fosdem, Feb 2025. Slides, Video, Link, LWN article 2024\nSeongJae Park, DAMON: Long-term Plans for Kernel That {Just Works,Extensible}. In Linux Kernel Memory Management Microconferenct at Linux Plumbers, Sep 2024. Slides, Video, Link SeongJae Park, DAMON Recipes: Ways to Save Memory Using a Linux Kernel Subsystem in the Real World. In Open Source Summit Europe, Sep 2024. Slides 1, Slides 2, Video, Link Jonathan Corbet, An update and future plans for DAMON. In Linux Weekly News, May 2024. Article SeongJae Park, DAMON Updates and Plans: Automation of DAMON tuning, tiering, and VM guest scaling. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, May 2024. Slides, Video, Link SeongJae Park, DAMO[N,S]?: Implementing Self-Driven Data Access-Aware Efficient Linux System. In Open Source Summit North America, Apr 2024. Slides, Video, Link 2023\nSeongJae Park, DAMON: Current Status and Future Plans. In Kernel Summit, Nov 2023. Slides, Video, Link SeongJae Park, Data Access Monitoring Operator (DAMO): User-Space Tool/Python Library for Access-Aware Profiling and Optimization of Your Linux Systems. In Open Source Summit Europe, Sep 2023. Slides, Video, Link Jonathan Corbet, A 2023 DAMON update. In Linux Weekly News, May 2023. Article SeongJae Park, DAMON, DAMOS, and DAMO: Kernel Subsystems and User-Space Tools for Data Access-Aware System Analysis/Optimizations. In Open Source Summit North America, May 2023. Slides, Video, Link SeongJae Park, DAMON updates and future plans. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, May 2023. Slides, Video, Link 2022\nSeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article SeongJae Park, Current Status, Future Plans, and Possible Collaborations for DAMON. In The Linux Kernel Summit, September 2022. Link SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster 2021\nSeongJae Park, Writing a fine-grained access pattern oriented lightweight kernel module using DAMON/DAMOS in 10 minutes. In The Linux Kernel Summit, September 2021. Slides, Video, Link Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article 2020\nSeongJae Park, DAMON: Data Access Monitoring Framework for Fun and Memory Management Optimizations, In The Linux Kernel Summit, August 2020. Slides, Video, Link Yunjae Lee, Yunhee Kim, and Heon. Y. Yeom, Lightweight Memory Tracing for Hot Data Identification, In Cluster computing, 2020. Paper Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article 2019\nSeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper SeongJae Park, Tracing Data Access Pattern with Bounded Overhead and Best-effort Accuracy. In The Linux Kernel Summit, September 2019. Slides, Link SeongJae Park, Yunjae Lee, Yunhee Kim, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Bounded Overhead and Accuracy. In IEEE International Workshop on Foundations and Applications of Self-* Systems (FAS* 2019), June 2019. Paper ","permalink":"https://damonitor.github.io/posts/damon_publications_talks/","summary":"This post provides lists of publications and presentations that cover DAMON.\nFeatured Publications and Talks There are quite amount of publications and talks. Below are featured ones for people who unsure what among the resources to use.\nAcademic papers For people who more familiar to academic papers, DAMON papers for Middleware'19 Industry and HPDC'22 are recommended to read and/or cite. The paper for Middleware'19 covers DAMON\u0026rsquo;s monitoring mechanisms and access pattern profiling-guided optimizations.","title":"DAMON Publications and Presentations"},{"content":"Below is a list of news around DAMON project.\nThis list is not exhaustive but just a DAMON maintainer\u0026rsquo;s collection of news. If you find a news that should be added to this list, please let us know at sj@kernel.org and/or damon@lists.linux.dev.\n2026 2026-02-17: LSF/MM/BPF topic for extending DAMON for per-CPUs/threads/reads/writes monitoring is proposed.\n2026-02-16: DAMON yearly retrospect for 2024 has posted.\n2026-02-10: An LSF/MM/BPF topic proposal for DAMON-based access-aware Transparent Hugepages is posted.\n2026-02-10: DAMON has nominated as one of Linux kernel ML library demonstration targets.\n2026-01-18: DAMON user-space tool will start command line auto-completion support.\n2025 2025-12-29: DAMON user-space tool v3.0.9 is released with support of trace-cmd for access pattern recording.\n2025-12-08: Third RFC patch series for per-CPU/threads/read/write monitoring is posted.\n2025-12-05: DAMON changes for Linux 6.19-rc1 has merged as a part of MM subsystem pull rquest. Major changes include but not limited to:\nPer-memcg per-node memory usage based DAMOS auto-tuning (patch series). This allows cgroup-level NUMA memory management, such as hot pages promotion, cold pages demotion and reclaim. This was developed as a collaboration with my now-ex-colleagues at Meta. Address alignment fix for DAMON modules (patch series). This was developed as a followup fix of ARM32 LAPE support, by Quanmin from Huawei. Pin-point targets removal (patch series). This was developed as a collaboration, as a followup of his vaddr-based DAMOS-migration to multi-destination nodes. 2025-11-13: An idea for DAMON-based CXL memory management aiming both bandwidth and capacity efficiency, which motivated by recent works from Micron and SK Hynix, has shared.\n2025-10-13: DAMON_STAT is build-enabled on Debian kernels.\n2025-10-03: DAMON changes including below two have been merged into Linux 6.18-rc1, via memory management subsystem pull request.\nvirtual address space page level monitoring support, which was developed by Yueyang Pan (Meta). 32-bit ARM LAPE support, which was collaboratively developed by Meta and Huawei people (Quanmin Yan and Ze Zuo). 2025-09-23: DAMON has presented at Kernel Recipes.\n2025-08-22: A patch series for making DAMON supports ARM (32bit) with LPAE has just landed on mm-new tree. It was made by a great and joyful collaboration between I and Huawei (Quanmin Yan and Zuo Ze).\n2025-08-10: DAMON user-space tool added a visualization script for cold memory tail visualization.\n2025-08-03: A prototype of per-CPUs and write-only monitoing is implemented and added to damon/next tree, and an experimental support of it is added to DAMON user-space tool (damo).\n2025-07-08: Bijan\u0026rsquo;s DAMON patch series for dynamic NUMA memory weighted interleaving that shows ~25% performance improvements on a test is now merged into mm-new.\n2025-08-22: A patch series for making DAMON supports ARM (32bit) with LPAE has just landed on mm-new tree. It was made by a great and joyful collaboration between I and Huawei (Quanmin Yan and Zuo Ze).\n2025-08-10: DAMON user-space tool added a visualization script for cold memory tail visualization.\n2025-08-03: A prototype of per-CPUs and write-only monitoing is implemented and added to damon/next tree, and an experimental support of it is added to DAMON user-space tool (damo).\n2025-07-08: Bijan\u0026rsquo;s DAMON patch series for dynamic NUMA memory weighted interleaving that shows ~25% performance improvements on a test is now merged into mm-new.\n2025-06-22: An ISCA'25 paper for better memory tiering is published. The paper uses DAMON and masim for showing access patterns of artificial and realistic workloads.\n2025_06_20: Bijan from Micron found DAMON-based post-allocation memory interleaving achieves approximately 25% speedup.\n2025-06-15: DAMON user-space tool added access temperature-sorted heatmap visualization feature. Example output of the feature are as below.\n2025-06-14: Intel has published another excellent ArXiv paper for memory tiering. The research used DAMON for a validation of the behavior of their approach (GPAC).\n2025-06-02: Changes for enabling CONFIG_DAMON by default has been merged into the mainline, by the second MM subsystem pull request for Linux 6.16-rc1. Later, the change has reverted by Linus Torvalds with his good explanation of the reason.\n2025-05-31: DAMON patches for more self-driven memory tiering have been merged into the mainline.\n2025-05-26: DAMON patches for simple and practical access monitoring have been posted\n2025-05-23: DAMON talk for Kernel Recipes 2025 has been updated.\n2025-05-12: RFC for build-enabling DAMON by default has been posted.\n2025-04-30: Researchers found automatically tuning parameters can improve memory tieirng performance by 2x, using DAMON and SK hynix\u0026rsquo; DAMON-based memory tiering solution as a part of their research.\n2025-04-24: Some of OpenSuse kernels will apparently build-enable DAMON in near future.\n2025-04-19: DAMON user-space tool feature for let users program their access pattern visualization in Python code, as a script or interactively on the Python interpreter, is developed.\n2025-04-17: Yet another DAMON-citation from a HCDS'25 paper is found. The paper discusses h/w-based CXL coherency management.\n2025-04-10: LWN made an excellent summary of the two DAMON sessions that we had at LSFMM+BPF 2025.\n2025-04-07: DAMON talk for OSSummit North America 2025 has been accepted and scheduled.\n2025-04-01: An EuroSys'25 paper for proactive demotion on tiered memory managment is published. It also shares evaluations of a DAMON-based tiering approach that is being used by HMSDK.\n2025-04-01: MM pull request for Linux 6.15-rc1 including below major DAMON changes is merged.\nMonitoring intervals auto-tuning Extending DAMOS filter types for hugepage, LRU-[in]active page, and [un]mapped pages DAMOS allow filters behavior improvement Important cleanups and fixes of code and documents 2025-03-20: DAMON session on LSF/MM/BPF 2025 has been scheduled.\n2025-03-19: RFC patch series for self-tuned DAMON-based memory tiering has posted with an evaluation result, and introduced by Phoronix.\n2025-03-03: DAMON monitoring intervals auto-tuning patch series has been posted, and queued in mm tree.\n2025-02-22: An academic paper showing DAMON-based memory tiering can be further improved using h/w-assisted promotion has been published.\n2025-02-12: DAMON monitoring intervals auto-tuning RFC patch series has been posted.\n2025-02-08: FOSDEM'25 DAMON talk record video has been available.\n2025-01-26: MM pull request for Linux 6.14-rc1 is merged with DAMON changes including below six patch series.\nmm/damon: add sample modules (link) mm/damon: replace most damon_callback usages in sysfs with new core functions (link) mm/damon: enable page level properties based monitoring (link) mm/damon: remove DAMON debugfs interface (link) mm/damon: extend DAMOS filters for inclusion (link) Docs/mm/damon: add tuning guide and misc updates (link) 2025-01-02: Two LSF/MM/BPF topic proposals for DAMON have been posted( 1, 2).\n2025-01-02: DAMON quaterly news letter for Q4 2024 has been posted\n2025-01-01: An LSF/MM/BPF topic proposal for gathering DAMON requirements for future MM has been posted.\n2025-01-01: DAMON debugfs interface removal patch series have been posted\n2024 204-12-26: RFC patch series for inclusive DAMOS filters has been posted\n2024-12-23: damo v2.6.1 is released with page level properties based monitoring support. Show a blog post for more details.\n2024-12-18: RFC patch series for page level properties based acces monitoing has been posted.\n2024-12-12: A DAMON presentation proposal has accepted to FOSDEM'25.\n2024-12-10: DAMON sample modules have posted.\n2024-12-02: DAMON monitoring parameters tuning guide example on a real server workload has been shared.\n2024-12-02: On Middlewar'24, a paper describing DAMON as a common cloud workload and evaluate their system for DAMON usage has presented.\n2024-11-25: damo v2.5.7 is releasd with tempearture-based regions filtering and formatting features. Show a blog post showing the details of the fetures and examples on a server workload.\n2024-11-18: damo v2.5.6 is released with heatmap snapshot visualization format and multiple kdamonds edit feature.\n2024-11-08: A guide to DAMON tuning and results interpretation for hot pages has posted.\n2024-11-04: damo v2.5.4 is released with recency/temperature histogram visualization features.\n2024-10-21: Monthly PyPI downloads of DAMON user-space tool, DAMO, doubled (8,000 -\u0026gt; 16,000) again after about ten dasys. 2024-10-18: DAMON projet site started hosting its own blog.\n2024-10-15: DAMON debugfs interface removal RFC patch has posted.\n2024-10-10: Monthly PyPI downloads of DAMON user-space too, DAMO, doubled again after ten days. 2024-10-08: Videos for DAMON recipes at Open Source Summit EU'2024 and DAMON long-term plans at Kernel Memory Management Microconference'2024 are uploaded to YouTube.\n2024-10-01: 2024-Q3 DAMON news letter including news for new features, more users, repos reorganizations, and conference talks is posted.\n2024-09-30: DAMON User Space Tool, DAMO, surpasses 4,000 monthly PyPI downloads! 2024-09-20: Livestreamed video for DAMON talk at kernel memory management microconference 2024 is now available at Youtube.\n2024-09-19: An academic paper preprint that optimizing THP using DAMON and BPF, titled \u0026ldquo;eBPF-mm: Userspace-guided memory management in Linux with eBPF\u0026rdquo; is uploaded to ArXiv\n2024-09-16: CONFIG_DAMON is enabled on Debian kernel\n2024-08-14: GitHub repos for non-kernel parts of DAMON project including \u0026lsquo;damo\u0026rsquo;, \u0026lsquo;damon-tests\u0026rsquo; and \u0026lsquo;damoos\u0026rsquo; will be moved from \u0026lsquo;awslabs\u0026rsquo; to \u0026lsquo;damonitor\u0026rsquo;, by 2024-09-05\n2024-07-29: VLDB paper about Aurora Serverless V2, which reveals their usage of DAMON on the product, is now available.\n2024-07-21: Memory Management subsystem pull request for Linux v6.11-rc1 is posted with DAMON changes for CXL memory tiering, documentation of a mailing tool for newcomers, and minor fixups.\n2024-07-18: DAMON topic for Linux Kernel Memory Management Microconference at LPC'24 has accepted.\n2024-07-11: ATC'24 also published two DAMON-citing papers at the same time. The first one proposes a way to improve monitoring accuracy of DAMON, while the second one mentions DAMON can be useful for extensible memory management.\n2024-07-11: A couple of OSDI'24 papers (1, 2) for memory tiering that references and exploring DAMON as a part of them are available now.\n2024-07-01: DAMON Quaterly Newsletter for 2024-Q2 has posted.\n2024-06-21: HacKerMaiL (hkml) has announced as a mailing tool for DAMON community that the maintainer is committed to support.\n2024-06-14: DAMON talk for Kernel Summit 2024 is proposed.\n2024-06-14: SK hynix\u0026rsquo; patch series \u0026ldquo;DAMON based tiered memory management for CXL memory\u0026rdquo; has merged into -mm tree.\n2024-06-12: DAMON talk for OpenSource Summit Europe 2024 has been accepted and scheduled.\n2024-05-18: Memory management subsystem pull request for Linux 6.10-rc1 has been posted. This pull request includes DAMOS young page filter, a DAMOS functionality kselftest, and misc cleanups/fixups for code, documentation, and tests.\n2024-05-17: LWN published an article introducing DAMON session at LSFMM 2024.\n2024-05-02: LSFMMBPF schedule is uploaded. DAMON talk is scheduled for the Monday noon.\n2024-04-29: The video of the DAMON presentation at OSSummit NA'24 is uploaded.\n2024-04-28: Yet another academic paper exploring DAMON for serverless computing has published on ASPLOS 24.\n2024-04-17: The third in-person DAMON meetup has held as a unconference session of Open Source Summint North America 2024\n2024-04-03: Oracle released a tool that helps finding distros that enabled DAMON\n2024-04-01: DAMO v2.2.8 is out. This version supports recording memory footprint of monitoring target processes together with their access pattern. Users could know when how much memory is allocated and really accessed. Such visualization is one of the future works, though.\n2024-03-13: Memory management subsystem pull request for Linux 6.9-rc1 has been posted. To quote Andrew’s summary for DAMON part:\nMore DAMON work from SeongJae Park in the series \u0026ldquo;mm/damon: make DAMON debugfs interface deprecation unignorable\u0026rdquo; \u0026ldquo;selftests/damon: add more tests for core functionalities and corner cases\u0026rdquo; \u0026ldquo;Docs/mm/damon: misc readability improvements\u0026rdquo; \u0026ldquo;mm/damon: let DAMOS feeds and tame/auto-tune itself\u0026rdquo; 2024-03-06: LSF/MM/BPF 2024 DAMON discussion topic is accepted\n2024-03-04: DAMO v2.2.4 is released with a new feature for access pattern-based profiling. For example, users can know which code is making their program\u0026rsquo;s memory usage unexpectedly high, or which code is intensively accessing memory, and optimize.\n2024-02-27: DAMON user-space tool, DAMO, has downloaded from PyPI more than 2,000 times last month. 2024-02-21: Yet another academic paper exploring DAMON for tiered memory management will be presented at EuroSys 2024\n2023-02-20: DAMO v2.2.2 is released with a new command, replay. It will hopefully help reproducing the real-world memory access pattern for analysis and benchmarks.\n2024-02-14: DAMON talk for OSSummit NA 2024 has been accepted and scheduled\n2024-02-09: DAMON in Amazon Linux 5.10.y kernel has updated to that of v6.7 Linux kernel. Major updates on this change include\nDAMOS apply target regions tracing and Sampling interval granularity monitoring results generation and DAMOS. 2024-01-29: LSF/MM/BPF 2024 topic proposal for DAMON has posted\n2024-01-15: SK Hynix shared their DAMOS-based tiered memory management test results showing 4-17% performance slowdown reduction, with patches for that.\n2024-01-12: LWN introduced the feedback-driven DAMOS aggressiveness auto-tuning as one of interesting changes for Linux v6.8\n2024-01-08: Memory management subsystem pull request for Linux v6.8-rc1 has posted. To quote Andrew\u0026rsquo;s summary for DAMON part:\nDAMON/DAMOS feature and maintenance work from SeongJae Park in the series \u0026ldquo;mm/damon: let users feed and tame/auto-tune DAMOS\u0026rdquo; \u0026ldquo;selftests/damon: add Python-written DAMON functionality tests\u0026rdquo; \u0026ldquo;mm/damon: misc updates for 6.8\u0026rdquo; 2023 2023-12-31: A retrospect of DAMON development in 2023 for the upstream community has posted.\n2023-12-27: SK Hynix released Heterogeneous Memory Software Development Kit (HMSDK) v2.0 which utilizes DAMON for tiered memory management.\n2023-11-24: A paper exploring DAMON and finding grateful areas to improve has uploaded to arXiv.\n2023-11-17: Livestreamed video for DAMON talk at kernel summit 2023 is now available at YouTube.\n2023-11-12: RFC idea for DAMOS-based tiered memory management is sent.\n2023-11-12: RFC idea for Access/Contiguity-aware Memory Auto-scaling is sent.\n2023-11-12: RFC patchset for Aim-oriented Feedback-driven DAMOS Aggressiveness Auto Tuning is sent.\n2023-11-08: The second in-person DAMON community meetup at LPC has accepted and announced.\n2023-11-02: Memory management subsystem pull requests for Linux v6.7-rc1, which contains the changes for DAMON, has sent.\n2023-10-22: An SOSP paper for tiered memory management that referencing and exploring DAMON has found.\n2023-10-12: LPC BoF session proposal for DAMON community in-person meetup has submitted.\n2023-10-04: DAMON talk for Kernel Summit track of Linux Plumbers Conference 2023 has accepted.\n2023-09-07: Yet another academic paper preprint regarding serverless on CXL using/citing DAMON has uploaded. The title of the preprint is \u0026ldquo;Understanding and Optimizing Serverless Workloads in CXL-Enabled Tiered Memory\u0026rdquo;\n2023-08-09: DAMON community started running its CI test against all stable kernels and report the results.\n2023-08-08: DAMON user-space tool (damo) has reached 100 Github stars. 2023-08-07: DAMON user-space tool (damo) has released its 100th version. A mail for the news, release stats, and appreciation to the DAMON community has also posted.\n2023-08-03: DAMON continuous functionality testing started testing stable rc kernels and report back the results.\n2023-08-01: DAMON Beer/Coffee/Tea meeting will be postponed from mid of August to end of OSSummit Euro 2023.\n2023-07-26: The kernel summit talk proposal for DAMON status and future plans has posted\n2023-07-10: Hocus wrote an article introducing DAMON as a kernel feature that could be useful for memory efficient VM, with its limitations.\n2023-06-30: DAMON talk for OSSummit EU 2023 has accepted and scheduled\n2023-06-25: DAMON userspace tool, damo has packaged for Debian/Ubuntu in addition to Fedora. It also turned out it was already packaged for ArchLinux. Refer to repology for detail.\n2023-05-26: Open Source Summit North America DAMON talk video is now available at Youtube\n2023-05-26: LSF/MM+BPF DAMON discussion video is now available at Youtube\n2023-05-17: Now DAMON user space tool (DAMO) is available at Fedora Packages\n2023-05-16: Michel from Fedora community is gonna package DAMON user space tool (DAMO) for Fedora!\n2023-05-16: An LWN article for LSF/MM+BPF DAMON discussion has uploaded\n2023-05-04: The schedule for DAMON talk/discussion at LSFMM is available now.\n2023-03-14: The schedule for DAMON talk at OSSummit NA is available now.\n2023-03-10: A DAMON talk proposal for OSSummit NA has accepted.\n2023-03-06: DAMOS filters feature has introduced as one of the most significant changes for Linux v6.3 by an LWN article\n2023-02-24: A preprint of an academic paper that compares their approach against DAMON has uploaded to ArXiv.\n2023-02-13: LSF/MM/BPF topic proposal for DAMON has posted\n2023-02-09: DAMON debugfs deprecation patchset has posted\n2022 2022-12-29: DAMON development summary of 2022 has shared and featured by Phoronix.\n2022-12-16: The DAMOS filtering for anon pages and/or memory cgroups have merged in mm tree.\n2022-10-19: An RFC patchset for efficient query-like DAMON monitoring results have posted\n2022-09-15: The video for my kernel summit DAMON talk this year is now available at Youtube\n2022-09-09: The plan for the first in-person DAMON community meetup at LPC and the in-person office hour at OSSummit EU has announced\n2022-09-06: AL2 5.10 kernel\u0026rsquo;s DAMON code has updated to that of v5.19\n2022-08-30: AL2 5.10 kernel\u0026rsquo;s DAMON code has updated to that of v5.18\n2022-08-22: LWN introduced DAMON-based LRU-lists manipulation (DAMON_LRU_SORT) in detail\n2022-08-15: LWN introduced DAMON’s new features including \u0026lsquo;LRU_SORT\u0026rsquo; as significant changes for Linux 6.0\n2022-08-12: Bi-weekly DAMON Beer/Coffee/Tea Chat series for open, regular, and informal community syncups and discussions has announced.\n2022-07-29: Current status, future plans, and possible collaborations for DAMON will be presented at the Kernel Summit 2022.\n2022-06-26: The poster of the DAOS paper is available online.\n2022-06-13: DAMON-based LRU-lists sorting patchset has posted and immediately merged in the -mm tree\n2022-05-04: A paper introducing DAMON and related works have accepted by HPDC22\n2022-05-03: Now DAMON has its own open mailing list\n2022-04-29: Patches for DAMON online tuning have merged in -mm tree\n2022-04-27: Android has backported and enabled building DAMON and DAMON_RECLAIM for the common kernel.\n2022-04-27: Alibaba has shared thier own DAMON user space tool.\n2022-02-28: The DAMON sysfs interface patchset has merged in -mm tree.\n2022-02-17: An RFC patchset for sysfs-based DAMON\u0026rsquo;s new user interface has posted.\n2022-01-20: A roadmap of DAMON has shared.\n2022-01-09: Linux 5.16 is released. \u0026ldquo;DAMON-based proactive memory reclamation, operation schemes and physical memory monitoring\u0026rdquo; are marked as prominent features of the release by the Kernel newbies and LWN.\n2021 2021-12-23: A great blog post for DAMON-enabled kernel has uploaded\n2021-11-07: DAMON patches for automated memory management optimization, the physical address space monitoring support, and proactive reclamation have merged in the mainline.\n2021-11-01: DAMON has released with Linux v5.15.\n2021-10-14: DAMON_RECLAIM patchset is merged in the Amazon Linux 5.10.y kernel tree.\n2021-10-02: DAMOS patchset is merged in the -mm tree.\n2021-09-23: DAMON and DAMOS are presented in the kernel summit. Slides, Video, Link\n2021-09-16: DAMON development tree on kernel.org is created.\n2021-09-08: DAMON patchset is merged in the Linus Torvalds\u0026rsquo; tree, aka mainline\n2021-09-07: DAMON/DAMOS will be presented at the Kernel Summit 2021\n2021-08-31: DAMON user-space tool is uploaded to the official Python packages system, PyPi\n2021-08-06: DAMON patchset is merged in the -mm tree\n2021-07-27: LWN published a second article introducing DAMON patchset series\n2021-06-11: DAMON-based proactive reclamation RFC patchset has shared on the hackernews and introduced by a Phoronix article\n2021-05-31: DAMON-based proactive reclamation RFC patchset has posted\n2021-05-26: DAMON-enabled Amazon Linux 2 kernels have deployed to all users\n2021-05-07: DAMON has merged in the public source tree for Amazon Linux v5.4.y kernel\n2021-04-05: damo now supports heatmap visualization on the terminal\n2021-03-31: DAMON user-space tool (damo) is released as an individual open source project\n2021-03-19: DAMON has merged in the public source tree for Amazon Linux v5.10.y kernel\n2021-03-04: DAMON supports for two latest LTS kernels announced\n2021-03-03: DAMON is merged in v5.10 based public Amazon Linux kernel tree\n2021-02-25: An example usage of DAMON for profiling is shared\n2021-01-07: A runtime system for DAMON-based optimal operation scheme finding is released\n2020 2020-12-03: Further plans around DAMON is shared\n2020-11-17: A real-world user story of DAMON is shared\n2020-09-26: The tests package for DAMON is released under GPL v2 license\n2020-08-19: A demo video is available\n2020-08-05: DAMON will be presented at the Kernel Summit 2020\n2020-06-04: Physical Memory Monitoring is now available\n2020-05-18: DAMON showcase website is announced\n2020-05-13: DAMON official document is uploaded online\n2020-02-20: DAMON has introduced by an LWN article\n2020-02-10: The first RFC of Data Access Monitoring-based Operating Schemes (DAMOS) has posted to LKML\n2020-01-23: The RFC of DAMON has introduced by LWN\u0026rsquo;s \u0026lsquo;Kernel patches of interest\u0026rsquo;\n2020-01-20: The first RFC patchset of DAMON has posted to LKML\n","permalink":"https://damonitor.github.io/posts/damon_news/","summary":"Below is a list of news around DAMON project.\nThis list is not exhaustive but just a DAMON maintainer\u0026rsquo;s collection of news. If you find a news that should be added to this list, please let us know at sj@kernel.org and/or damon@lists.linux.dev.\n2026 2026-02-17: LSF/MM/BPF topic for extending DAMON for per-CPUs/threads/reads/writes monitoring is proposed.\n2026-02-16: DAMON yearly retrospect for 2024 has posted.\n2026-02-10: An LSF/MM/BPF topic proposal for DAMON-based access-aware Transparent Hugepages is posted.","title":"DAMON News List"},{"content":"This document helps you estimating the amount of benefit that you could get from DAMON-based system optimizations, and describes how you could achieve it.\nCheck The Signs No optimization can provide same extent of benefit to every case. Therefore you should first guess how much improvements you could get using DAMON. If some of below conditions match your situation, you could consider using DAMON.\nLow IPC and High Cache Miss Ratios. Low IPC means most of the CPU time is spent waiting for the completion of time-consuming operations such as memory access, while high cache miss ratios mean the caches don\u0026rsquo;t help it well. DAMON is not for cache level optimization, but DRAM level. However, improving DRAM management will also help this case by reducing the memory operation latency. Memory Over-commitment and Unknown Users. If you are doing memory overcommitment and you cannot control every user of your system, a memory bank run could happen at any time. You can estimate when it will happen based on DAMON\u0026rsquo;s monitoring results and act earlier to avoid or deal better with the crisis. Frequent Memory Pressure. Frequent memory pressure means your system has wrong configurations or memory hogs. DAMON will help you find the right configuration and/or the criminals. Heterogeneous Memory System. If your system is utilizing memory devices that placed between DRAM and traditional hard disks, such as non-volatile memory or fast SSDs, DAMON could help you utilizing the devices more efficiently. Profile If you found some positive signals, you could start by profiling your workloads using DAMON. Find major workloads on your systems and analyze their data access pattern to find something wrong or can be improved. The DAMON user space tool (damo) will be useful for this. You can get damo from https://github.com/damonitor/damo.\nWe recommend you to start from working set size distribution check using damo report wss. If the distribution is ununiform or quite different from what you estimated, you could consider Memory Configuration optimization.\nThen, review the overall access pattern in heatmap form using damo report heats. If it shows a simple pattern consists of a small number of memory regions having high contrast of access temperature, you could consider manual Program Modification.\nIf the access pattern is very frequently changing so that you cannot figure out what is the performance important region using your human eye, Automated DAMON-based Memory Operations might help the case owing to its machine-level microscope view.\nIf you still want to absorb more benefits, you should develop Personalized DAMON Application for your special case.\nYou don\u0026rsquo;t need to take only one approach among the above plans, but you could use multiple of the above approaches to maximize the benefit.\nOptimize If the profiling result also says it\u0026rsquo;s worth trying some optimization, you could consider below approaches. Note that some of the below approaches assume that your systems are configured with swap devices or other types of auxiliary memory so that you don\u0026rsquo;t strictly required to accommodate the whole working set in the main memory. Most of the detailed optimization should be made on your concrete understanding of your memory devices.\nMemory Configuration No more no less, DRAM should be large enough to accommodate only important working sets, because DRAM is highly performance critical but expensive and heavily consumes the power. However, knowing the size of the real important working sets is difficult. As a consequence, people usually equips unnecessarily large or too small DRAM. Many problems stem from such wrong configurations.\nUsing the working set size distribution report provided by damo report wss, you can know the appropriate DRAM size for you. For example, roughly speaking, if you worry about only 95 percentile latency, you don\u0026rsquo;t need to equip DRAM of a size larger than 95 percentile working set size.\nLet\u0026rsquo;s see a real example. This page \u0026lt;https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/guide.html#memory-configuration\u0026gt; shows the heatmap and the working set size distributions/changes of freqmine workload in PARSEC3 benchmark suite. The working set size spikes up to 180 MiB, but keeps smaller than 50 MiB for more than 95% of the time. Even though you give only 50 MiB of memory space to the workload, it will work well for 95% of the time. Meanwhile, you can save the 130 MiB of memory space.\nProgram Modification If the data access pattern heatmap plotted by damo report heats is quite simple so that you can understand how the things are going in the workload with your human eye, you could manually optimize the memory management.\nFor example, suppose that the workload has two big memory object but only one object is frequently accessed while the other one is only occasionally accessed. Then, you could modify the program source code to keep the hot object in the main memory by invoking mlock() or madvise() with MADV_WILLNEED. Or, you could proactively evict the cold object using madvise() with MADV_COLD or MADV_PAGEOUT. Using both together would be also worthy.\nA research work [1] using the mlock() achieved up to 2.55x performance speedup.\nLet\u0026rsquo;s see another realistic example access pattern for this kind of optimizations. This another page \u0026lt;https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/guide.html#program-modification\u0026gt; shows the visualized access patterns of streamcluster workload in PARSEC3 benchmark suite. We can easily identify the 100 MiB sized hot object.\nAutomated DAMON-based Memory Operations Though Manual Program Optimization works well in many cases and DAMON can help it, modifying the source code is not a good option in many cases. First of all, the source code could be too old or unavailable. And, many workloads will have complex data access patterns that even hard to distinguish hot memory objects and cold memory objects with the human eye. Finding the mapping from the visualized access pattern to the source code and injecting the hinting system calls inside the code will also be quite challenging.\nBy using DAMON-based operation schemes (DAMOS) via damo schemes, you will be able to easily optimize your workload in such a case. Our example schemes called \u0026rsquo;efficient THP\u0026rsquo; and \u0026lsquo;proactive reclamation\u0026rsquo; achieved significant speedup and memory space saves against 25 realistic workloads [2].\nThat said, note that you need careful tune of the schemes (e.g., target region size and age) and monitoring attributes for the successful use of this approach. Because the optimal values of the parameters will be dependent on each system and workload, misconfiguring the parameters could result in worse memory management.\nFor the tuning, you could measure the performance metrics such as IPC, TLB misses, and swap in/out events and adjusts the parameters based on their changes. The total number and the total size of the regions that each scheme is applied, which are provided via the debugfs interface and the programming interface can also be useful. Writing a program automating this optimal parameter could be an option.\nPersonalized DAMON Application Above approaches will work well for many general cases, but would not enough for some special cases.\nIf this is the case, it might be the time to forget the comfortable use of the user space tool and dive into the debugfs interface (refer to :doc:usage for the detail) of DAMON. Using the interface, you can control the DAMON more flexibly. Therefore, you can write your personalized DAMON application that controls the monitoring via the debugfs interface, analyzes the result, and applies complex optimizations itself. Using this, you can make more creative and wise optimizations.\nIf you are a kernel space programmer, writing kernel space DAMON applications using the API (refer to the :doc:/mm/damon/api for more detail) would be an option.\nReference Practices Referencing previously done successful practices could help you getting the sense for this kind of optimizations. There is an academic paper [1] reporting the visualized access pattern and manual Program Modification results for a number of realistic workloads. You can also get the visualized access patterns [3,4,5] and Automated DAMON-based Memory Operations results for other realistic workloads that collected with latest version of DAMON [2].\n[1] https://dl.acm.org/doi/10.1145/3366626.3368125\n[2] https://damonitor.github.io/test/result/perf/latest/html/\n[3] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html\n[4] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html\n[5] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html\n","permalink":"https://damonitor.github.io/posts/damon_optimization_guide/","summary":"This document helps you estimating the amount of benefit that you could get from DAMON-based system optimizations, and describes how you could achieve it.\nCheck The Signs No optimization can provide same extent of benefit to every case. Therefore you should first guess how much improvements you could get using DAMON. If some of below conditions match your situation, you could consider using DAMON.\nLow IPC and High Cache Miss Ratios.","title":"DAMON-based System Optimization Guide"},{"content":"DAMON is lightweight. It increases system memory usage by 0.39% and slows target workloads down by 1.16%.\nDAMON is accurate and useful for memory management optimizations. An experimental DAMON-based operation scheme for THP, namely \u0026rsquo;ethp\u0026rsquo;, removes 76.15% of THP memory overheads while preserving 51.25% of THP speedup. Another experimental DAMON-based \u0026lsquo;proactive reclamation\u0026rsquo; implementation, namely \u0026lsquo;prcl\u0026rsquo;, reduces 93.38% of residential sets and 23.63% of system memory footprint while incurring only 1.22% runtime overhead in the best case (parsec3/freqmine).\nSetup On QEMU/KVM based virtual machines utilizing 130GB of RAM and 36 vCPUs hosted by AWS EC2 i3.metal instances that running a Linux v5.10 based kernel that v24 DAMON patchset is applied, I measure runtime and consumed system memory while running various realistic workloads with several configurations. From each of PARSEC3 [3] and SPLASH-2X [4] benchmark suites I pick 12 workloads, so I use 24 workloads in total. I use another wrapper scripts [5] for convenient setup and run of the workloads.\nMeasurement For the measurement of the amount of consumed memory in system global scope, I drop caches before starting each of the workloads and monitor \u0026lsquo;MemFree\u0026rsquo; in the \u0026lsquo;/proc/meminfo\u0026rsquo; file. To make results more stable, I repeat the runs 5 times and average results.\nConfigurations The configurations I use are as below.\norig: Linux v5.10 with \u0026lsquo;madvise\u0026rsquo; THP policy rec: \u0026lsquo;orig\u0026rsquo; plus DAMON running with virtual memory access recording prec: \u0026lsquo;orig\u0026rsquo; plus DAMON running with physical memory access recording thp: same with \u0026lsquo;orig\u0026rsquo;, but use \u0026lsquo;always\u0026rsquo; THP policy ethp: \u0026lsquo;orig\u0026rsquo; plus a DAMON operation scheme, \u0026rsquo;efficient THP\u0026rsquo; prcl: \u0026lsquo;orig\u0026rsquo; plus a DAMON operation scheme, \u0026lsquo;proactive reclaim [6]\u0026rsquo; I use \u0026lsquo;rec\u0026rsquo; for measurement of DAMON overheads to target workloads and system memory. \u0026lsquo;prec\u0026rsquo; is for physical memory monitroing and recording. It monitors 17GB sized \u0026lsquo;System RAM\u0026rsquo; region. The remaining configs including \u0026rsquo;thp\u0026rsquo;, \u0026rsquo;ethp\u0026rsquo;, and \u0026lsquo;prcl\u0026rsquo; are for measurement of DAMON monitoring accuracy.\n\u0026rsquo;ethp\u0026rsquo; and \u0026lsquo;prcl\u0026rsquo; are simple DAMON-based operation schemes developed for proof of concepts of DAMON. \u0026rsquo;ethp\u0026rsquo; reduces memory space waste of THP [1,2], by using DAMON for the decision of promotions and demotion for huge pages, while \u0026lsquo;prcl\u0026rsquo; is as similar as the original work. For example, those can be implemented as below::\n# format: \u0026lt;min/max size\u0026gt; \u0026lt;min/max frequency (0-100)\u0026gt; \u0026lt;min/max age\u0026gt; \u0026lt;action\u0026gt; # ethp: Use huge pages if a region shows \u0026gt;=5% access rate, use regular # pages if a region \u0026gt;=2MB shows 0 access rate for \u0026gt;=7 seconds min max 5 max min max hugepage 2M max min min 7s max nohugepage # prcl: If a region \u0026gt;=4KB shows 0 access rate for \u0026gt;=5 seconds, page out. 4K max 0 0 5s max pageout Note that these examples are designed with my only straightforward intuition because those are for only proof of concepts and monitoring accuracy of DAMON. In other words, those are not for production. For production use, those should be more tuned. For automation of such tuning, you can use a user space tool called DAMOOS [8]. For the evaluation, we use \u0026rsquo;ethp\u0026rsquo; as same to above example, but we use DAMOOS-tuned \u0026lsquo;prcl\u0026rsquo; for each workload.\nThe evaluation is done using the tests package for DAMON, damon-tests [7]. Using it, you can do the evaluation and generate a report on your own.\n[1] \u0026ldquo;Redis latency problems troubleshooting\u0026rdquo;, https://redis.io/topics/latency\n[2] \u0026ldquo;Disable Transparent Huge Pages (THP)\u0026rdquo;, https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/\n[3] \u0026ldquo;The PARSEC Becnhmark Suite\u0026rdquo;, https://parsec.cs.princeton.edu/index.htm\n[4] \u0026ldquo;SPLASH-2x\u0026rdquo;, https://parsec.cs.princeton.edu/parsec3-doc.htm#splash2x\n[5] \u0026ldquo;parsec3_on_ubuntu\u0026rdquo;, https://github.com/sjp38/parsec3_on_ubuntu\n[6] \u0026ldquo;Proactively reclaiming idle memory\u0026rdquo;, https://lwn.net/Articles/787611/\n[7] \u0026ldquo;damon-tests\u0026rdquo;, https://github.com/damonitor/damon-tests\n[8] \u0026ldquo;DAMOOS\u0026rdquo;, https://github.com/damonitor/damoos\nResults Below two tables show the measurement results. The runtimes are in seconds while the memory usages are in KiB. Each configuration except \u0026lsquo;orig\u0026rsquo; shows its overhead relative to \u0026lsquo;orig\u0026rsquo; in percent within parenthesizes.::\nruntime orig rec (overhead) prec (overhead) thp (overhead) ethp (overhead) prcl (overhead) parsec3/blackscholes 139.658 140.168 (0.37) 139.385 (-0.20) 138.367 (-0.92) 139.279 (-0.27) 147.024 (5.27) parsec3/bodytrack 123.788 124.622 (0.67) 123.636 (-0.12) 125.115 (1.07) 123.840 (0.04) 141.928 (14.65) parsec3/canneal 207.491 210.318 (1.36) 217.927 (5.03) 174.287 (-16.00) 202.609 (-2.35) 225.483 (8.67) parsec3/dedup 18.292 18.301 (0.05) 18.378 (0.47) 18.264 (-0.15) 18.298 (0.03) 20.541 (12.30) parsec3/facesim 343.893 340.286 (-1.05) 338.217 (-1.65) 332.953 (-3.18) 333.840 (-2.92) 365.650 (6.33) parsec3/fluidanimate 339.959 326.886 (-3.85) 330.286 (-2.85) 331.239 (-2.57) 326.011 (-4.10) 341.684 (0.51) parsec3/freqmine 445.987 436.332 (-2.16) 435.946 (-2.25) 435.704 (-2.31) 437.595 (-1.88) 451.414 (1.22) parsec3/raytrace 184.106 182.158 (-1.06) 182.056 (-1.11) 183.180 (-0.50) 183.545 (-0.30) 202.197 (9.83) parsec3/streamcluster 599.990 674.091 (12.35) 617.314 (2.89) 521.864 (-13.02) 551.971 (-8.00) 696.127 (16.02) parsec3/swaptions 220.462 222.637 (0.99) 220.449 (-0.01) 219.921 (-0.25) 221.607 (0.52) 223.956 (1.59) parsec3/vips 87.767 88.700 (1.06) 87.461 (-0.35) 87.466 (-0.34) 87.875 (0.12) 91.768 (4.56) parsec3/x264 110.843 117.856 (6.33) 113.023 (1.97) 108.665 (-1.97) 115.434 (4.14) 117.811 (6.29) splash2x/barnes 131.441 129.275 (-1.65) 128.341 (-2.36) 119.317 (-9.22) 126.199 (-3.99) 147.602 (12.30) splash2x/fft 59.705 58.382 (-2.22) 58.858 (-1.42) 45.949 (-23.04) 59.939 (0.39) 64.548 (8.11) splash2x/lu_cb 132.552 131.604 (-0.72) 131.846 (-0.53) 132.320 (-0.18) 132.100 (-0.34) 140.289 (5.84) splash2x/lu_ncb 150.215 149.670 (-0.36) 149.646 (-0.38) 148.823 (-0.93) 149.416 (-0.53) 152.338 (1.41) splash2x/ocean_cp 84.033 76.405 (-9.08) 75.104 (-10.63) 73.487 (-12.55) 77.789 (-7.43) 77.380 (-7.92) splash2x/ocean_ncp 153.833 154.247 (0.27) 156.227 (1.56) 106.619 (-30.69) 139.299 (-9.45) 165.030 (7.28) splash2x/radiosity 143.566 143.654 (0.06) 142.426 (-0.79) 141.193 (-1.65) 141.740 (-1.27) 157.817 (9.93) splash2x/radix 49.984 49.996 (0.02) 50.519 (1.07) 46.573 (-6.82) 50.724 (1.48) 50.695 (1.42) splash2x/raytrace 133.238 134.337 (0.83) 134.389 (0.86) 134.833 (1.20) 131.073 (-1.62) 145.541 (9.23) splash2x/volrend 121.700 120.652 (-0.86) 120.560 (-0.94) 120.629 (-0.88) 119.581 (-1.74) 129.422 (6.35) splash2x/water_nsquared 370.771 375.236 (1.20) 376.829 (1.63) 355.592 (-4.09) 354.087 (-4.50) 419.606 (13.17) splash2x/water_spatial 133.295 132.931 (-0.27) 132.762 (-0.40) 133.090 (-0.15) 133.809 (0.39) 153.647 (15.27) total 4486.580 4538.750 (1.16) 4481.600 (-0.11) 4235.430 (-5.60) 4357.660 (-2.87) 4829.510 (7.64) memused.avg orig rec (overhead) prec (overhead) thp (overhead) ethp (overhead) prcl (overhead) parsec3/blackscholes 1828693.600 1834084.000 (0.29) 1823839.800 (-0.27) 1819296.600 (-0.51) 1830281.800 (0.09) 1603975.800 (-12.29) parsec3/bodytrack 1424963.400 1440085.800 (1.06) 1438384.200 (0.94) 1421718.400 (-0.23) 1432834.600 (0.55) 1439283.000 (1.00) parsec3/canneal 1036782.600 1052828.800 (1.55) 1050148.600 (1.29) 1035104.400 (-0.16) 1051145.400 (1.39) 1050019.400 (1.28) parsec3/dedup 2511841.400 2507374.000 (-0.18) 2472450.600 (-1.57) 2523557.600 (0.47) 2508912.000 (-0.12) 2493347.200 (-0.74) parsec3/facesim 537769.800 550740.800 (2.41) 548683.600 (2.03) 543547.800 (1.07) 560556.600 (4.24) 482782.600 (-10.23) parsec3/fluidanimate 570268.600 585598.000 (2.69) 579837.800 (1.68) 571433.000 (0.20) 582112.800 (2.08) 470073.400 (-17.57) parsec3/freqmine 982941.400 996253.200 (1.35) 993919.800 (1.12) 990531.800 (0.77) 1000994.400 (1.84) 750685.800 (-23.63) parsec3/raytrace 1737446.000 1749908.800 (0.72) 1741183.800 (0.22) 1726674.800 (-0.62) 1748530.200 (0.64) 1552275.600 (-10.66) parsec3/streamcluster 115857.000 155194.400 (33.95) 158272.800 (36.61) 122125.200 (5.41) 134545.600 (16.13) 133448.600 (15.18) parsec3/swaptions 13694.200 28451.800 (107.76) 28464.600 (107.86) 12797.800 (-6.55) 25328.200 (84.96) 28138.400 (105.48) parsec3/vips 2976126.400 3002408.600 (0.88) 3008218.800 (1.08) 2978258.600 (0.07) 2995428.600 (0.65) 2936338.600 (-1.34) parsec3/x264 3233886.200 3258790.200 (0.77) 3248355.000 (0.45) 3232070.000 (-0.06) 3256360.200 (0.69) 3254707.400 (0.64) splash2x/barnes 1210470.600 1211918.600 (0.12) 1204507.000 (-0.49) 1210892.800 (0.03) 1217414.800 (0.57) 944053.400 (-22.01) splash2x/fft 9697440.000 9604535.600 (-0.96) 9210571.800 (-5.02) 9867368.000 (1.75) 9637571.800 (-0.62) 9804092.000 (1.10) splash2x/lu_cb 510680.400 521792.600 (2.18) 517724.600 (1.38) 513500.800 (0.55) 519980.600 (1.82) 351787.000 (-31.11) splash2x/lu_ncb 512896.200 529353.600 (3.21) 521248.600 (1.63) 513493.200 (0.12) 523793.400 (2.12) 418701.600 (-18.37) splash2x/ocean_cp 3320800.200 3313688.400 (-0.21) 3225585.000 (-2.87) 3359032.200 (1.15) 3316591.800 (-0.13) 3304702.200 (-0.48) splash2x/ocean_ncp 3915132.400 3917401.000 (0.06) 3884086.400 (-0.79) 7050398.600 (80.08) 4532528.600 (15.77) 3920395.800 (0.13) splash2x/radiosity 1456908.200 1467611.800 (0.73) 1453612.600 (-0.23) 1466695.400 (0.67) 1467495.600 (0.73) 421197.600 (-71.09) splash2x/radix 2345874.600 2318202.200 (-1.18) 2261499.200 (-3.60) 2438228.400 (3.94) 2373697.800 (1.19) 2336605.600 (-0.40) splash2x/raytrace 43258.800 57624.200 (33.21) 55164.600 (27.52) 46204.400 (6.81) 60475.000 (39.80) 48865.400 (12.96) splash2x/volrend 149615.000 163809.400 (9.49) 162115.400 (8.36) 149119.600 (-0.33) 162747.800 (8.78) 157734.600 (5.43) splash2x/water_nsquared 40384.400 54848.600 (35.82) 53796.600 (33.21) 41455.800 (2.65) 53226.400 (31.80) 58260.600 (44.27) splash2x/water_spatial 670580.200 680444.200 (1.47) 670020.400 (-0.08) 668262.400 (-0.35) 678552.000 (1.19) 372931.000 (-44.39) total 40844300.000 41002900.000 (0.39) 40311600.000 (-1.30) 44301900.000 (8.47) 41671052.000 (2.02) 38334431.000 (-6.14) DAMON Overheads In total, DAMON virtual memory access recording feature (\u0026lsquo;rec\u0026rsquo;) incurs 1.16% runtime overhead and 0.39% memory space overhead. Even though the size of the monitoring target region becomes much larger with the physical memory access recording (\u0026lsquo;prec\u0026rsquo;), it still shows only modest amount of overhead (-0.11% for runtime and -1.30% for memory footprint).\nFor a convenient test run of \u0026lsquo;rec\u0026rsquo; and \u0026lsquo;prec\u0026rsquo;, I use a Python wrapper. The wrapper constantly consumes about 10-15MB of memory. This becomes a high memory overhead if the target workload has a small memory footprint. Nonetheless, the overheads are not from DAMON, but from the wrapper, and thus should be ignored. This fake memory overhead continues in \u0026rsquo;ethp\u0026rsquo; and \u0026lsquo;prcl\u0026rsquo;, as those configurations are also using the Python wrapper.\nEfficient THP THP \u0026lsquo;always\u0026rsquo; enabled policy achieves 5.60% speedup but incurs 8.47% memory overhead. It achieves 30.69% speedup in the best case, but 80.08% memory overhead in the worst case. Interestingly, both the best and worst-case are with \u0026lsquo;splash2x/ocean_ncp\u0026rsquo;).\nThe 2-lines implementation of data access monitoring based THP version (\u0026rsquo;ethp\u0026rsquo;) shows 2.87% speedup and 2.02% memory overhead. In other words, \u0026rsquo;ethp\u0026rsquo; removes 76.15% of THP memory waste while preserving 51.25% of THP speedup in total. In the case of the \u0026lsquo;splash2x/ocean_ncp\u0026rsquo;, \u0026rsquo;ethp\u0026rsquo; removes 80.30% of THP memory waste while preserving 30.79% of THP speedup.\nProactive Reclamation As similar to the original work, I use 4G \u0026lsquo;zram\u0026rsquo; swap device for this configuration. Also note that we use DAMOOS-tuned ethp schemes for each workload.\nIn total, our 1 line implementation of Proactive Reclamation, \u0026lsquo;prcl\u0026rsquo;, incurred 7.64% runtime overhead in total while achieving 6.14% system memory footprint reduction. Even in the worst case, the runtime overhead was only 16.02%.\nNonetheless, as the memory usage is calculated with \u0026lsquo;MemFree\u0026rsquo; in \u0026lsquo;/proc/meminfo\u0026rsquo;, it contains the SwapCached pages. As the swapcached pages can be easily evicted, I also measured the residential set size of the workloads::\nrss.avg orig rec (overhead) prec (overhead) thp (overhead) ethp (overhead) prcl (overhead) parsec3/blackscholes 587536.800 585720.000 (-0.31) 586233.400 (-0.22) 587045.400 (-0.08) 586753.400 (-0.13) 252207.400 (-57.07) parsec3/bodytrack 32302.200 32290.600 (-0.04) 32261.800 (-0.13) 32215.800 (-0.27) 32173.000 (-0.40) 6798.800 (-78.95) parsec3/canneal 842370.600 841443.400 (-0.11) 844012.400 (0.19) 838074.400 (-0.51) 841700.800 (-0.08) 840804.000 (-0.19) parsec3/dedup 1180414.800 1164634.600 (-1.34) 1188886.200 (0.72) 1207821.000 (2.32) 1193896.200 (1.14) 572359.200 (-51.51) parsec3/facesim 311848.400 311709.800 (-0.04) 311790.800 (-0.02) 317345.800 (1.76) 315443.400 (1.15) 188488.000 (-39.56) parsec3/fluidanimate 531868.000 531885.600 (0.00) 531828.800 (-0.01) 532988.000 (0.21) 532959.600 (0.21) 415153.200 (-21.94) parsec3/freqmine 552491.000 552718.600 (0.04) 552807.200 (0.06) 556574.200 (0.74) 554374.600 (0.34) 36573.400 (-93.38) parsec3/raytrace 879683.400 880752.200 (0.12) 879907.000 (0.03) 870631.000 (-1.03) 880952.200 (0.14) 293119.200 (-66.68) parsec3/streamcluster 110991.800 110937.200 (-0.05) 110964.600 (-0.02) 115606.800 (4.16) 116199.000 (4.69) 110108.200 (-0.80) parsec3/swaptions 5665.000 5718.400 (0.94) 5720.600 (0.98) 5682.200 (0.30) 5628.600 (-0.64) 3613.800 (-36.21) parsec3/vips 32143.600 31823.200 (-1.00) 31912.200 (-0.72) 33164.200 (3.18) 33925.800 (5.54) 27813.600 (-13.47) parsec3/x264 81534.000 81811.000 (0.34) 81708.400 (0.21) 83052.400 (1.86) 83758.800 (2.73) 81691.800 (0.19) splash2x/barnes 1220515.200 1218291.200 (-0.18) 1217699.600 (-0.23) 1228551.600 (0.66) 1220669.800 (0.01) 681096.000 (-44.20) splash2x/fft 9915850.400 10036461.000 (1.22) 9881242.800 (-0.35) 10334603.600 (4.22) 10006993.200 (0.92) 8975181.200 (-9.49) splash2x/lu_cb 511327.200 511679.000 (0.07) 511761.600 (0.08) 511971.600 (0.13) 511711.200 (0.08) 338005.000 (-33.90) splash2x/lu_ncb 511505.000 506816.800 (-0.92) 511392.800 (-0.02) 496623.000 (-2.91) 511410.200 (-0.02) 404734.000 (-20.87) splash2x/ocean_cp 3398834.000 3405017.800 (0.18) 3415287.800 (0.48) 3443604.600 (1.32) 3416264.200 (0.51) 3387134.000 (-0.34) splash2x/ocean_ncp 3947092.800 3939805.400 (-0.18) 3952311.600 (0.13) 7165858.800 (81.55) 4610075.000 (16.80) 3944753.400 (-0.06) splash2x/radiosity 1475024.000 1474053.200 (-0.07) 1475032.400 (0.00) 1483718.800 (0.59) 1475919.600 (0.06) 99637.200 (-93.25) splash2x/radix 2431302.200 2416928.600 (-0.59) 2455596.800 (1.00) 2568526.400 (5.64) 2479966.800 (2.00) 2437406.600 (0.25) splash2x/raytrace 23274.400 23278.400 (0.02) 23287.200 (0.05) 28828.000 (23.86) 27800.200 (19.45) 5667.000 (-75.65) splash2x/volrend 44106.800 44151.400 (0.10) 44186.000 (0.18) 45200.400 (2.48) 44751.200 (1.46) 16912.000 (-61.66) splash2x/water_nsquared 29427.200 29425.600 (-0.01) 29402.400 (-0.08) 28055.400 (-4.66) 28572.400 (-2.90) 13207.800 (-55.12) splash2x/water_spatial 664312.200 664095.600 (-0.03) 663025.200 (-0.19) 664100.600 (-0.03) 663597.400 (-0.11) 261214.200 (-60.68) total 29321300.000 29401500.000 (0.27) 29338300.000 (0.06) 33179900.000 (13.16) 30175600.000 (2.91) 23393600.000 (-20.22) In total, 20.22% of residential sets were reduced.\nWith parsec3/freqmine, \u0026lsquo;prcl\u0026rsquo; reduced 93.38% of residential sets and 23.63% of system memory usage while incurring only 1.22% runtime overhead.\n","permalink":"https://damonitor.github.io/posts/damon_evaluation/","summary":"DAMON is lightweight. It increases system memory usage by 0.39% and slows target workloads down by 1.16%.\nDAMON is accurate and useful for memory management optimizations. An experimental DAMON-based operation scheme for THP, namely \u0026rsquo;ethp\u0026rsquo;, removes 76.15% of THP memory overheads while preserving 51.25% of THP speedup. Another experimental DAMON-based \u0026lsquo;proactive reclamation\u0026rsquo; implementation, namely \u0026lsquo;prcl\u0026rsquo;, reduces 93.38% of residential sets and 23.63% of system memory footprint while incurring only 1.22% runtime overhead in the best case (parsec3/freqmine).","title":"DAMON Evaluation"},{"content":"A summary of DAMON development in 2022 has posted: https://lore.kernel.org/damon/20221229171209.162356-1-sj@kernel.org/\n2022 was a year of active and healthy DAMON development.\nSeven new DAMON major features were delivered to users. Some of those were featured in articles and academic papers.\nIt was possible thanks to the DAMON community. The community has expanded with its own mailing list and an open bi-weekly chat series. 40 people contributed their great code to DAMON via making their 275 commits merged into the mainline. About 33% of the commits were made by Amazon-external contributors.\nThe amount of DAMON changes in 2022 (v5.15..v6.2-rc1) was not that tiny compared to other subsystems. About 0.2% of the commits for whole Linux tree was for DAMON. Among the changes for DAMON\u0026rsquo;s parent subsystem, mm, about 8% of commits and 14% of lines of changes were made for DAMON.\n","permalink":"https://damonitor.github.io/posts/damon_stat_2022/","summary":"A summary of DAMON development in 2022 has posted: https://lore.kernel.org/damon/20221229171209.162356-1-sj@kernel.org/\n2022 was a year of active and healthy DAMON development.\nSeven new DAMON major features were delivered to users. Some of those were featured in articles and academic papers.\nIt was possible thanks to the DAMON community. The community has expanded with its own mailing list and an open bi-weekly chat series. 40 people contributed their great code to DAMON via making their 275 commits merged into the mainline.","title":"Summary of DAMON Development in 2022"},{"content":"I realized I didn\u0026rsquo;t introduce a good, intuitive example use case of DAMON[0] for profiling so far, though DAMON is not for only profiling. One straightforward and realistic usage of DAMON as a profiling tool would be recording the monitoring results with callstack and visualize those by timeline together.\nFor example, below shows that visualization for a realistic workload, namely \u0026lsquo;fft\u0026rsquo; in SPLASH-2X benchmark suite. The upper-most graph shows how DAMON-detected working set size of the workload (y-axis) changes by time (x-axis). The middle graph shows when (x-axis) which address range of the memory (y-axis) has how frequency accessed (color). The lower-most graph, finally, shows when (x-axis) what function the workload was executing, and by what function call chain it was called.\nFrom this, you can know there are three memory access bursting phases in the workload and FFT1DOnce.cons::prop.2() looks responsible for the first and second hot phase, while Transpose() is responsible for the last one. Now the programmer can take a deep look in the functions and optimize the code (e.g., adding madvise() or mlock() calls).\nWe used the mlock()-based optimization approach to a range of other realistic benchmark workloads. The optimized versions achieved up to about 2.5x performance improvement under memory pressure[1].\nNote: I made the uppermost two figures of above \u0026lsquo;fft\u0026rsquo; visualization (working set size and access frequency of each memory region by time) via the DAMON user space tool[2], while the lowermost one (callstack by time) is made using perf and speedscope[3]. We have no descent and totally automated tool for that yet (will be implemented soon, maybe under perf as a perf-script[4]), but you could reproduce that with below commands.\n$ # run the workload $ sudo damo record $(pidof \u0026lt;the workload\u0026gt;) \u0026amp; $ sudo perf record -g --pid $(pidof \u0026lt;the workload\u0026gt;) $ # after your workload finished (you should also finish perf on your own) $ damo report wss --sortby time --plot wss.pdf $ damo report heats --heatmap freq.pdf $ sudo perf script | speedscope - $ # open wss.pdf and freq.pdf with our favorite pdf viewer [0] https://damonitor.github.io\n[1] https://linuxplumbersconf.org/event/4/contributions/548/attachments/311/590/damon_ksummit19.pdf\n[2] https://lore.kernel.org/linux-mm/20201215115448.25633-8-sjpark@amazon.com/\n[3] https://www.speedscope.app/\n[4] https://lore.kernel.org/linux-mm/20210107120729.22328-1-sjpark@amazon.com/\n","permalink":"https://damonitor.github.io/posts/damon_profile_callstack_example/","summary":"I realized I didn\u0026rsquo;t introduce a good, intuitive example use case of DAMON[0] for profiling so far, though DAMON is not for only profiling. One straightforward and realistic usage of DAMON as a profiling tool would be recording the monitoring results with callstack and visualize those by timeline together.\nFor example, below shows that visualization for a realistic workload, namely \u0026lsquo;fft\u0026rsquo; in SPLASH-2X benchmark suite. The upper-most graph shows how DAMON-detected working set size of the workload (y-axis) changes by time (x-axis).","title":"An example of DAMON usage for profiling"},{"content":"DAMON contains a number of tests based on the kselftest and kunit in its patchset itself. As it is preferred to contain only tests having short runtime in kernel tree, I organized time consuming tests in a package and used it in my company only. Tests could be used as a good document and essential for contributors. For the reason, I promised I will make it open source in the last kernel summit talk (https://linuxplumbersconf.org/event/7/contributions/659/).\nYesterday, I finally publicly released the package under GPL v2. Now you can use the package to understand the DAMON interface and test it on your machine by yourself.\n","permalink":"https://damonitor.github.io/posts/damon-tests_open_sourced/","summary":"DAMON contains a number of tests based on the kselftest and kunit in its patchset itself. As it is preferred to contain only tests having short runtime in kernel tree, I organized time consuming tests in a package and used it in my company only. Tests could be used as a good document and essential for contributors. For the reason, I promised I will make it open source in the last kernel summit talk (https://linuxplumbersconf.","title":"Tests package for DAMON is released under GPL v2"},{"content":"A DAMON showcase website[1] is open. There are\nthe official documentation of DAMON[2], the heatmap format dynamic access pattern of various realistic workloads for heap area[3], mmap()-ed area[4], and stack[5] area, the dynamic working set size distribution[6] and chronological working set size changes[7], and the latest performance test results[8]. [1] https://damonitor.github.io\n[2] https://damonitor.github.io/doc/html/latest\n[3] https://damonitor.github.io/test/result/visual/latest/heatmap.0.html\n[4] https://damonitor.github.io/test/result/visual/latest/heatmap.1.html\n[5] https://damonitor.github.io/test/result/visual/latest/heatmap.2.html\n[6] https://damonitor.github.io/test/result/visual/latest/wss_sz.html\n[7] https://damonitor.github.io/test/result/visual/latest/wss_time.html\n[8] https://damonitor.github.io/test/result/perf/latest/html/index.html\n","permalink":"https://damonitor.github.io/posts/damon_github_page/","summary":"A DAMON showcase website[1] is open. There are\nthe official documentation of DAMON[2], the heatmap format dynamic access pattern of various realistic workloads for heap area[3], mmap()-ed area[4], and stack[5] area, the dynamic working set size distribution[6] and chronological working set size changes[7], and the latest performance test results[8]. [1] https://damonitor.github.io\n[2] https://damonitor.github.io/doc/html/latest\n[3] https://damonitor.github.io/test/result/visual/latest/heatmap.0.html\n[4] https://damonitor.github.io/test/result/visual/latest/heatmap.1.html\n[5] https://damonitor.github.io/test/result/visual/latest/heatmap.2.html\n[6] https://damonitor.github.io/test/result/visual/latest/wss_sz.html\n[7] https://damonitor.github.io/test/result/visual/latest/wss_time.html\n[8] https://damonitor.github.io/test/result/perf/latest/html/index.html","title":"Opening a Showcase Website for DAMON"},{"content":"DAMON is a Linux kernel subsystem for efficient data access monitoring and access-aware system operations.\nWith increasingly data-intensive workloads and limited DRAM capacity, optimal memory management based on dynamic access patterns is becoming increasingly important. Such mechanisms are only possible if accurate and efficient dynamic access pattern monitoring is available.\nDAMON is a Linux kernel subsystem for such data access monitoring and access-aware system operations. It is designed with its key access monitroing mechanisms and a major feature called DAMOS, that make it\naccurate (for DRAM level memory management), light-weight (enough to be applied online in production), scalable (keeps above properties regardless of memory size) and automated (tuning and access-aware memory maangement operations). Simple mechanisms based on DAMOS can achieve up to 12% performance improvement and 91% memory savings. Detailed evaluation of DAMON and DAMON-based system optimizations are available at another post.\nDAMON is being used in real-world products including AWS Aurora Serverless and SK hynix HMSDK for proactive reclamation and CXL memory tiering. A number of academic researches are also utilizing DAMON for profiling and prototyping, as show by citations of the two (1, 2) DAMON intro papers.\nDAMON is available on Linux mainline since v5.15, and multiple major distros including Alma, Amazon, Android, Arch, CentOS, Debian, Fedora, open SUSE, Oracle.\nDemo Video/Screenshots Demo Video Demo Screenshot Recent News Below are only a short list of recent news. For complete list of the news, please refer to a dedicated post.\n2026-02-17: LSF/MM/BPF topic for extending DAMON for per-CPUs/threads/reads/writes monitoring is proposed.\n2026-02-16: DAMON yearly retrospect for 2024 has posted.\n2026-02-10: An LSF/MM/BPF topic proposal for DAMON-based access-aware Transparent Hugepages is posted.\n2026-02-10: DAMON has nominated as one of Linux kernel ML library demonstration targets.\n2026-01-18: DAMON user-space tool will start command line auto-completion support.\nGetting Started You can start using DAMON by\ninstalling its user-space tool and following the tutorial of the user-space tool. Further read official documents to better understand DAMON\u0026rsquo;s design and usages. If you prefer more informal videos, slides, or papers, those from past DAMON presentations are also available.\nDepending on your system and your desired use case, you might need to\ninstall DAMON-enabled and newer kernel, and/or run the automated tests suite. You can also participate in DAMON development.\nUser-space Tool A user-space tool for DAMON, which is called DAMO (Data Access Monitoring Operator), is available at Github, PyPi and Linux distro package systems. You may start using DAMON by following the Getting Started of the tool for start.\nOfficial Document The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Those for next major release are also available(users/sysadmins doc, kernel programmers doc).\nPublications and Presentations Below are featured publications and presentations covering DAMON. For full list of the past and upcoming presentations, please refer to another dedicated post.\nAcademic papers For people who more familiar to academic papers, DAMON papers for Middleware'19 Industry and HPDC'22 are recommended to read and/or cite. The paper for Middleware'19 covers DAMON\u0026rsquo;s monitoring mechanisms and access pattern profiling-guided optimizations. The paper for HPDC'22 extends the coverage to DAMOS (automated access-aware system operations) and user-space driven auto-tuning.\nTalks for beginners and users If you are looking for a resources to start with, below talks are recommended.\nSeongJae Park, Page-level and Fleet-wide Data Access Monitoring for Meta. In Linux Plumbers Refereed Track, Dec 2025. Slides, Video, Link SeongJae Park, Overcoming Observer Effects in Memory Management with DAMON. In Kernel Recipes, Sep 2025. Slides, Video, Link SeongJae Park, DAMON: Kernel Subsystem for Data Access Monitoring and Access-aware System Operations. In Fosdem, Feb 2025. Slides, Video, Link Talks for experts and developers If you want to track recent DAMON developmeent status and plans, below talks are recommended.\nSeongJae Park, DAMON-based Pages Migration for {C,G,X}PU [un]attached NUMA nodes. In Device and Specific PurposeMemory MC at Linux Plumbers, Dec 2025. slides, Video, Link SeongJae Park, DAMON Requirements for Access-aware MM of Future. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, Mar 2025. Slides, Link SeongJae Park, DAMON Updates and Plans: Monitoring Parameters Auot-tuning and Memory Tiering. In Linux Storage | Filesystem | MM \u0026amp; BPF Summit, Mar 2025. Slides, Link Community DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.\nThe community is mainly driven by the mailing list (damon@lists.linux.dev). All the patches are posted there and reviewed. Almost every discussions are also made there.\nFor easy communication, a mailing tool called HacKerMaiL is recommended. The tool is maintained by DAMON maintainer and committed to support DAMON community.\nIf you prefer web-based interface for reading the mails, you can use lore. If you prefer the traditional subscription based mailing workflow, you can subscribe to the mailing list via subspace.kernel.org following the instruction.\nThe community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.\nContribution DAMON and its related projects including damo and hackermail are open source projects. You can contribute to the development following the guidelines. Please refer to below contribution guidelines of each project to further look into the process.\nDAMON damo hkml Setup for Advanced Use Cases Install DAMON is a part of Linux kernel. To use DAMON, therefore, you should ensure your system is running with a DAMON-enabled Linux kernel.\nTo check if your kernel is a DAMON-enabled one, you could:\n$ if grep -q '^CONFIG_DAMON=y' /boot/config-$(uname -r); then echo \u0026quot;installed\u0026quot; else echo \u0026quot;not installed\u0026quot; fi As of 2025-11-25, kernels of major Linux distros including Alma Linux, Amazon Linux, Android, Arch, CentOS, Debian, Fedora, openSUSE and Oracle Linux enabled DAMON. If your distro provides an appropriate DAMON-enabled kernel, install it using the package manager of the distro.\nIf your package system doesn\u0026rsquo;t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree, build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:\n$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE $ make olddefconfig $ echo 'CONFIG_DAMON=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_VADDR=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_PADDR=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_SYSFS=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_DBGFS=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_RECLAIM=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_LRU_SORT=y' \u0026gt;\u0026gt; ./.config $ echo 'CONFIG_DAMON_STAT=y' \u0026gt;\u0026gt; ./.config $ make -j$(nproc) $ sudo make modules_install install Source Code There are several Linux kernel source trees having DAMON for different users. You may pick one among those based on your needs.\nFor users who want a stable version of DAMON, Linus Torvalds' mainline tree or stable kernels are recommended. Note that DAMON has merged into mainline since v5.15.\nIf you have interests in DAMON features under development, refer to developement source trees section of DAMON official documents.\nTests Package There is a tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.\nSo, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org or damon@lists.linux.dev) or github.\nShowcase Website There is a showcase website that you can get more information of DAMON. The site hosts\nthe heatmap format dynamic access pattern of various realistic workloads for heap area, mmap()-ed area, and stack area, the dynamic working set size distribution and chronological working set size changes, and daily performance test results. Good to Read Evaluation Results Evaluation of DAMON and DAMON-based system optimizations are available.\nDAMON-based System Optimization Guide A guide for DAMON-based system optimizations are also available.\nProfile-Guided Optimization Example An example of DAMON-based profile-guided optimization is also available.\n","permalink":"https://damonitor.github.io/posts/damon/","summary":"DAMON is a Linux kernel subsystem for efficient data access monitoring and access-aware system operations.\nWith increasingly data-intensive workloads and limited DRAM capacity, optimal memory management based on dynamic access patterns is becoming increasingly important. Such mechanisms are only possible if accurate and efficient dynamic access pattern monitoring is available.\nDAMON is a Linux kernel subsystem for such data access monitoring and access-aware system operations. It is designed with its key access monitroing mechanisms and a major feature called DAMOS, that make it","title":"DAMON: Data Access Monitor"}]