| .. SPDX-License-Identifier: GPL-2.0 | |
| ================================== | |
| Netmem Support for Network Drivers | |
| ================================== | |
| This document outlines the requirements for network drivers to support netmem, | |
| an abstract memory type that enables features like device memory TCP. By | |
| supporting netmem, drivers can work with various underlying memory types | |
| with little to no modification. | |
| Benefits of Netmem : | |
| * Flexibility: Netmem can be backed by different memory types (e.g., struct | |
| page, DMA-buf), allowing drivers to support various use cases such as device | |
| memory TCP. | |
| * Future-proof: Drivers with netmem support are ready for upcoming | |
| features that rely on it. | |
| * Simplified Development: Drivers interact with a consistent API, | |
| regardless of the underlying memory implementation. | |
| Design Principles | |
| ================= | |
| Memory providers (or the default ``page_pool`` allocator) allocate underlying | |
| memory (``struct net_iov`` or ``struct page``), cast it to ``netmem_ref``, and | |
| supply it to ``page_pool``. The ``page_pool``, drivers, and networking stack | |
| operate on ``netmem_ref`` as the abstract type. Existing ``page_pool`` APIs | |
| that allocate or free ``struct page`` are legacy compatibility wrappers for | |
| drivers that do not yet support ``netmem_ref``. Code that is not yet | |
| ``netmem``-aware should be converted to ``netmem_ref`` unless it will never | |
| need to support ``netmem``. | |
| 1. **Operate on netmem_ref, do not downcast**: ``page_pool``, drivers, and the | |
| core networking stack should deal with ``netmem_ref`` rather than | |
| ``struct net_iov`` or ``struct page``. Downcasting ``netmem_ref`` to | |
| ``struct net_iov`` or ``struct page`` is not allowed unless a code path | |
| strictly cannot function without knowing the underlying memory type (for | |
| example, ``kmap_local_page()``). In those cases, to keep call sites simple, | |
| add a ``netmem`` helper that performs the operation on behalf of the caller, | |
| cleanly handles all ``net_iov`` and ``page`` cases, and returns an error if | |
| the ``netmem`` type cannot support the requested operation. | |
| 2. **Decouple memory providers from net_iov**: Memory providers are not | |
| architecturally limited to ``struct net_iov``; a memory provider that returns | |
| ``struct page``-backed ``netmem_ref``\ s to upper layers is allowed. Today, | |
| in-tree memory providers only supply ``struct net_iov`` and some existing | |
| code still reflects that limitation, but new code must not assume that using | |
| a memory provider implies ``net_iov`` memory and should, as much as possible, | |
| generalize existing limitations to match the design principles. | |
| 3. **Decouple net_iov from unreadability**: ``struct net_iov`` is flexible and | |
| has no inherent restrictions; it may represent either CPU-readable or | |
| unreadable memory. Today, in-tree ``net_iov`` implementations are unreadable | |
| by the CPU (``netmem_address()`` returns ``NULL``) and some existing code | |
| still reflects that limitation, but new code must not assume ``net_iov`` | |
| implies unreadable memory (check readability via ``netmem_address()`` or | |
| ``skb_frags_readable()`` instead) and should, as much as possible, generalize | |
| existing limitations to match the design principles. | |
| 4. **Delegate complexity to the lowest layer**: Each layer must respect its | |
| abstraction boundary. ``page_pool`` must not implement per-memory-provider | |
| custom logic in its main code; instead, it delegates provider-specific | |
| handling to ``struct memory_provider_ops``. Similarly, core networking code | |
| should avoid per-``netmem``-type branching and instead delegate operations | |
| to ``netmem`` helpers that handle the underlying memory type. | |
| 5. **Do not mix skb fragment memory sources**: All ``frags[]`` in an | |
| ``sk_buff`` must either be ``struct page``-backed or belong to the same | |
| memory provider instance (for example, the same ``devmem`` binding or | |
| ``io_uring`` area). Mixing ``struct page`` and memory-provider fragments, | |
| or mixing fragments from different memory providers within a single | |
| ``sk_buff``, is not allowed (including when coalescing ``sk_buff``\ s). | |
| Driver RX Requirements | |
| ====================== | |
| 1. The driver must support page_pool. | |
| 2. The driver must support the tcp-data-split ethtool option. | |
| 3. The driver must use the page_pool netmem APIs for payload memory. The netmem | |
| APIs currently 1-to-1 correspond with page APIs. Conversion to netmem should | |
| be achievable by switching the page APIs to netmem APIs and tracking memory | |
| via netmem_refs in the driver rather than struct page * : | |
| - page_pool_alloc -> page_pool_alloc_netmem | |
| - page_pool_get_dma_addr -> page_pool_get_dma_addr_netmem | |
| - page_pool_put_page -> page_pool_put_netmem | |
| Not all page APIs have netmem equivalents at the moment. If your driver | |
| relies on a missing netmem API, feel free to add and propose to netdev@, or | |
| reach out to the maintainers and/or almasrymina@google.com for help adding | |
| the netmem API. | |
| 4. The driver must use the following PP_FLAGS: | |
| - PP_FLAG_DMA_MAP: netmem is not dma-mappable by the driver. The driver | |
| must delegate the dma mapping to the page_pool, which knows when | |
| dma-mapping is (or is not) appropriate. | |
| - PP_FLAG_DMA_SYNC_DEV: netmem dma addr is not necessarily dma-syncable | |
| by the driver. The driver must delegate the dma syncing to the page_pool, | |
| which knows when dma-syncing is (or is not) appropriate. | |
| - PP_FLAG_ALLOW_UNREADABLE_NETMEM. The driver must specify this flag iff | |
| tcp-data-split is enabled. | |
| 5. The driver must not assume the netmem is readable and/or backed by pages. | |
| The netmem returned by the page_pool may be unreadable, in which case | |
| netmem_address() will return NULL. The driver must correctly handle | |
| unreadable netmem, i.e. don't attempt to handle its contents when | |
| netmem_address() is NULL. | |
| Ideally, drivers should not have to check the underlying netmem type via | |
| helpers like netmem_is_net_iov() or convert the netmem to any of its | |
| underlying types via netmem_to_page() or netmem_to_net_iov(). In most cases, | |
| netmem or page_pool helpers that abstract this complexity are provided | |
| (and more can be added). | |
| 6. The driver must use page_pool_dma_sync_netmem_for_cpu() in lieu of | |
| dma_sync_single_range_for_cpu(). For some memory providers, dma_syncing for | |
| CPU will be done by the page_pool, for others (particularly dmabuf memory | |
| provider), dma syncing for CPU is the responsibility of the userspace using | |
| dmabuf APIs. The driver must delegate the entire dma-syncing operation to | |
| the page_pool which will do it correctly. | |
| 7. Avoid implementing driver-specific recycling on top of the page_pool. Drivers | |
| cannot hold onto a struct page to do their own recycling as the netmem may | |
| not be backed by a struct page. However, you may hold onto a page_pool | |
| reference with page_pool_fragment_netmem() or page_pool_ref_netmem() for | |
| that purpose, but be mindful that some netmem types might have longer | |
| circulation times, such as when userspace holds a reference in zerocopy | |
| scenarios. | |
| Driver TX Requirements | |
| ====================== | |
| 1. The Driver must not pass the netmem dma_addr to any of the dma-mapping APIs | |
| directly. This is because netmem dma_addrs may come from a source like | |
| dma-buf that is not compatible with the dma-mapping APIs. | |
| Helpers like netmem_dma_unmap_page_attrs() & netmem_dma_unmap_addr_set() | |
| should be used in lieu of dma_unmap_page[_attrs](), dma_unmap_addr_set(). | |
| The netmem variants will handle netmem dma_addrs correctly regardless of the | |
| source, delegating to the dma-mapping APIs when appropriate. | |
| Not all dma-mapping APIs have netmem equivalents at the moment. If your | |
| driver relies on a missing netmem API, feel free to add and propose to | |
| netdev@, or reach out to the maintainers and/or almasrymina@google.com for | |
| help adding the netmem API. | |
| 2. Driver should declare support by setting `netdev->netmem_tx` to the | |
| appropriate mode: | |
| - `NETMEM_TX_DMA`: for physical devices that perform DMA. | |
| - `NETMEM_TX_NO_DMA`: for virtual or passthrough devices that do | |
| not DMA, but still support handling of netmem-backed skbs. |