1
linux/Documentation/networking/page_pool.rst

200 lines
6.6 KiB
ReStructuredText
Raw Normal View History

.. SPDX-License-Identifier: GPL-2.0
=============
Page Pool API
=============
.. kernel-doc:: include/net/page_pool/helpers.h
:doc: page_pool allocator
Architecture overview
=====================
.. code-block:: none
+------------------+
| Driver |
+------------------+
^
|
|
|
v
+--------------------------------------------+
| request memory |
+--------------------------------------------+
^ ^
| |
| Pool empty | Pool has entries
| |
v v
+-----------------------+ +------------------------+
| alloc (and map) pages | | get page from cache |
+-----------------------+ +------------------------+
^ ^
| |
| cache available | No entries, refill
| | from ptr-ring
| |
v v
+-----------------+ +------------------+
| Fast cache | | ptr-ring cache |
+-----------------+ +------------------+
Monitoring
==========
Information about page pools on the system can be accessed via the netdev
genetlink family (see Documentation/netlink/specs/netdev.yaml).
API interface
=============
The number of pools created **must** match the number of hardware queues
unless hardware restrictions make that impossible. This would otherwise beat the
purpose of page pool, which is allocate pages fast from cache without locking.
This lockless guarantee naturally comes from running under a NAPI softirq.
The protection doesn't strictly have to be NAPI, any guarantee that allocating
a page will cause no race conditions is enough.
.. kernel-doc:: net/core/page_pool.c
:identifiers: page_pool_create
.. kernel-doc:: include/net/page_pool/types.h
:identifiers: struct page_pool_params
.. kernel-doc:: include/net/page_pool/helpers.h
:identifiers: page_pool_put_page page_pool_put_full_page
page_pool_recycle_direct page_pool_free_va
page_pool_dev_alloc_pages page_pool_dev_alloc_frag
page_pool_dev_alloc page_pool_dev_alloc_va
page_pool_get_dma_addr page_pool_get_dma_dir
.. kernel-doc:: net/core/page_pool.c
:identifiers: page_pool_put_page_bulk page_pool_get_stats
DMA sync
--------
Driver is always responsible for syncing the pages for the CPU.
Drivers may choose to take care of syncing for the device as well
or set the ``PP_FLAG_DMA_SYNC_DEV`` flag to request that pages
allocated from the page pool are already synced for the device.
If ``PP_FLAG_DMA_SYNC_DEV`` is set, the driver must inform the core what portion
of the buffer has to be synced. This allows the core to avoid syncing the entire
page when the drivers knows that the device only accessed a portion of the page.
Most drivers will reserve headroom in front of the frame. This part
of the buffer is not touched by the device, so to avoid syncing
it drivers can set the ``offset`` field in struct page_pool_params
appropriately.
For pages recycled on the XDP xmit and skb paths the page pool will
use the ``max_len`` member of struct page_pool_params to decide how
much of the page needs to be synced (starting at ``offset``).
When directly freeing pages in the driver (page_pool_put_page())
the ``dma_sync_size`` argument specifies how much of the buffer needs
to be synced.
If in doubt set ``offset`` to 0, ``max_len`` to ``PAGE_SIZE`` and
pass -1 as ``dma_sync_size``. That combination of arguments is always
correct.
Note that the syncing parameters are for the entire page.
This is important to remember when using fragments (``PP_FLAG_PAGE_FRAG``),
where allocated buffers may be smaller than a full page.
Unless the driver author really understands page pool internals
it's recommended to always use ``offset = 0``, ``max_len = PAGE_SIZE``
with fragmented page pools.
Stats API and structures
------------------------
If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
page_pool_get_stats() and structures described below are available.
It takes a pointer to a ``struct page_pool`` and a pointer to a struct
page_pool_stats allocated by the caller.
Older drivers expose page pool statistics via ethtool or debugfs.
The same statistics are accessible via the netlink netdev family
in a driver-independent fashion.
.. kernel-doc:: include/net/page_pool/types.h
:identifiers: struct page_pool_recycle_stats
struct page_pool_alloc_stats
struct page_pool_stats
Coding examples
===============
Registration
------------
.. code-block:: c
/* Page pool registration */
struct page_pool_params pp_params = { 0 };
struct xdp_rxq_info xdp_rxq;
int err;
pp_params.order = 0;
/* internal DMA mapping in page_pool */
pp_params.flags = PP_FLAG_DMA_MAP;
pp_params.pool_size = DESC_NUM;
pp_params.nid = NUMA_NO_NODE;
pp_params.dev = priv->dev;
page_pool: allow caching from safely localized NAPI Recent patches to mlx5 mentioned a regression when moving from driver local page pool to only using the generic page pool code. Page pool has two recycling paths (1) direct one, which runs in safe NAPI context (basically consumer context, so producing can be lockless); and (2) via a ptr_ring, which takes a spin lock because the freeing can happen from any CPU; producer and consumer may run concurrently. Since the page pool code was added, Eric introduced a revised version of deferred skb freeing. TCP skbs are now usually returned to the CPU which allocated them, and freed in softirq context. This places the freeing (producing of pages back to the pool) enticingly close to the allocation (consumer). If we can prove that we're freeing in the same softirq context in which the consumer NAPI will run - lockless use of the cache is perfectly fine, no need for the lock. Let drivers link the page pool to a NAPI instance. If the NAPI instance is scheduled on the same CPU on which we're freeing - place the pages in the direct cache. With that and patched bnxt (XDP enabled to engage the page pool, sigh, bnxt really needs page pool work :() I see a 2.6% perf boost with a TCP stream test (app on a different physical core than softirq). The CPU use of relevant functions decreases as expected: page_pool_refill_alloc_cache 1.17% -> 0% _raw_spin_lock 2.41% -> 0.98% Only consider lockless path to be safe when NAPI is scheduled - in practice this should cover majority if not all of steady state workloads. It's usually the NAPI kicking in that causes the skb flush. The main case we'll miss out on is when application runs on the same CPU as NAPI. In that case we don't use the deferred skb free path. Reviewed-by: Tariq Toukan <tariqt@nvidia.com> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> Tested-by: Dragos Tatulea <dtatulea@nvidia.com> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-04-12 21:26:04 -07:00
pp_params.napi = napi; /* only if locking is tied to NAPI */
pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
page_pool = page_pool_create(&pp_params);
err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0);
if (err)
goto err_out;
err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool);
if (err)
goto err_out;
NAPI poller
-----------
.. code-block:: c
/* NAPI Rx poller */
enum dma_data_direction dma_dir;
dma_dir = page_pool_get_dma_dir(dring->page_pool);
while (done < budget) {
if (some error)
page_pool_recycle_direct(page_pool, page);
if (packet_is_xdp) {
if XDP_DROP:
page_pool_recycle_direct(page_pool, page);
} else (packet_is_skb) {
skb_mark_for_recycle(skb);
new_page = page_pool_dev_alloc_pages(page_pool);
}
}
Stats
-----
.. code-block:: c
#ifdef CONFIG_PAGE_POOL_STATS
/* retrieve stats */
struct page_pool_stats stats = { 0 };
if (page_pool_get_stats(page_pool, &stats)) {
/* perhaps the driver reports statistics with ethool */
ethtool_print_allocation_stats(&stats.alloc_stats);
ethtool_print_recycle_stats(&stats.recycle_stats);
}
#endif
Driver unload
-------------
.. code-block:: c
/* Driver unload */
page_pool_put_full_page(page_pool, page, false);
xdp_rxq_info_unreg(&xdp_rxq);