2018-07-20 02:01:47 -07:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Common functionality of grant device.
|
|
|
|
|
*
|
|
|
|
|
* Copyright (c) 2006-2007, D G Murray.
|
|
|
|
|
* (c) 2009 Gerd Hoffmann <kraxel@redhat.com>
|
|
|
|
|
* (c) 2018 Oleksandr Andrushchenko, EPAM Systems Inc.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _GNTDEV_COMMON_H
|
|
|
|
|
#define _GNTDEV_COMMON_H
|
|
|
|
|
|
|
|
|
|
#include <linux/mm.h>
|
|
|
|
|
#include <linux/mman.h>
|
|
|
|
|
#include <linux/mmu_notifier.h>
|
|
|
|
|
#include <linux/types.h>
|
2020-03-23 09:15:11 -07:00
|
|
|
#include <xen/interface/event_channel.h>
|
xen/gntdev: Avoid blocking in unmap_grant_pages()
unmap_grant_pages() currently waits for the pages to no longer be used.
In https://github.com/QubesOS/qubes-issues/issues/7481, this lead to a
deadlock against i915: i915 was waiting for gntdev's MMU notifier to
finish, while gntdev was waiting for i915 to free its pages. I also
believe this is responsible for various deadlocks I have experienced in
the past.
Avoid these problems by making unmap_grant_pages async. This requires
making it return void, as any errors will not be available when the
function returns. Fortunately, the only use of the return value is a
WARN_ON(), which can be replaced by a WARN_ON when the error is
detected. Additionally, a failed call will not prevent further calls
from being made, but this is harmless.
Because unmap_grant_pages is now async, the grant handle will be sent to
INVALID_GRANT_HANDLE too late to prevent multiple unmaps of the same
handle. Instead, a separate bool array is allocated for this purpose.
This wastes memory, but stuffing this information in padding bytes is
too fragile. Furthermore, it is necessary to grab a reference to the
map before making the asynchronous call, and release the reference when
the call returns.
It is also necessary to guard against reentrancy in gntdev_map_put(),
and to handle the case where userspace tries to map a mapping whose
contents have not all been freed yet.
Fixes: 745282256c75 ("xen/gntdev: safely unmap grants in case they are still in use")
Cc: stable@vger.kernel.org
Signed-off-by: Demi Marie Obenour <demi@invisiblethingslab.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/r/20220622022726.2538-1-demi@invisiblethingslab.com
Signed-off-by: Juergen Gross <jgross@suse.com>
2022-06-21 19:27:26 -07:00
|
|
|
#include <xen/grant_table.h>
|
2018-07-20 02:01:47 -07:00
|
|
|
|
2018-07-20 02:01:48 -07:00
|
|
|
struct gntdev_dmabuf_priv;
|
|
|
|
|
|
2018-07-20 02:01:47 -07:00
|
|
|
struct gntdev_priv {
|
|
|
|
|
/* Maps with visible offsets in the file descriptor. */
|
|
|
|
|
struct list_head maps;
|
|
|
|
|
/* lock protects maps and freeable_maps. */
|
|
|
|
|
struct mutex lock;
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_XEN_GRANT_DMA_ALLOC
|
|
|
|
|
/* Device for which DMA memory is allocated. */
|
|
|
|
|
struct device *dma_dev;
|
|
|
|
|
#endif
|
2018-07-20 02:01:48 -07:00
|
|
|
|
|
|
|
|
#ifdef CONFIG_XEN_GNTDEV_DMABUF
|
|
|
|
|
struct gntdev_dmabuf_priv *dmabuf_priv;
|
|
|
|
|
#endif
|
2018-07-20 02:01:47 -07:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct gntdev_unmap_notify {
|
|
|
|
|
int flags;
|
|
|
|
|
/* Address relative to the start of the gntdev_grant_map. */
|
|
|
|
|
int addr;
|
2020-03-23 09:15:11 -07:00
|
|
|
evtchn_port_t event;
|
2018-07-20 02:01:47 -07:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct gntdev_grant_map {
|
xen/gntdev: Accommodate VMA splitting
Prior to this commit, the gntdev driver code did not handle the
following scenario correctly with paravirtualized (PV) Xen domains:
* User process sets up a gntdev mapping composed of two grant mappings
(i.e., two pages shared by another Xen domain).
* User process munmap()s one of the pages.
* User process munmap()s the remaining page.
* User process exits.
In the scenario above, the user process would cause the kernel to log
the following messages in dmesg for the first munmap(), and the second
munmap() call would result in similar log messages:
BUG: Bad page map in process doublemap.test pte:... pmd:...
page:0000000057c97bff refcount:1 mapcount:-1 \
mapping:0000000000000000 index:0x0 pfn:...
...
page dumped because: bad pte
...
file:gntdev fault:0x0 mmap:gntdev_mmap [xen_gntdev] readpage:0x0
...
Call Trace:
<TASK>
dump_stack_lvl+0x46/0x5e
print_bad_pte.cold+0x66/0xb6
unmap_page_range+0x7e5/0xdc0
unmap_vmas+0x78/0xf0
unmap_region+0xa8/0x110
__do_munmap+0x1ea/0x4e0
__vm_munmap+0x75/0x120
__x64_sys_munmap+0x28/0x40
do_syscall_64+0x38/0x90
entry_SYSCALL_64_after_hwframe+0x61/0xcb
...
For each munmap() call, the Xen hypervisor (if built with CONFIG_DEBUG)
would print out the following and trigger a general protection fault in
the affected Xen PV domain:
(XEN) d0v... Attempt to implicitly unmap d0's grant PTE ...
(XEN) d0v... Attempt to implicitly unmap d0's grant PTE ...
As of this writing, gntdev_grant_map structure's vma field (referred to
as map->vma below) is mainly used for checking the start and end
addresses of mappings. However, with split VMAs, these may change, and
there could be more than one VMA associated with a gntdev mapping.
Hence, remove the use of map->vma and rely on map->pages_vm_start for
the original start address and on (map->count << PAGE_SHIFT) for the
original mapping size. Let the invalidate() and find_special_page()
hooks use these.
Also, given that there can be multiple VMAs associated with a gntdev
mapping, move the "mmu_interval_notifier_remove(&map->notifier)" call to
the end of gntdev_put_map, so that the MMU notifier is only removed
after the closing of the last remaining VMA.
Finally, use an atomic to prevent inadvertent gntdev mapping re-use,
instead of using the map->live_grants atomic counter and/or the map->vma
pointer (the latter of which is now removed). This prevents the
userspace from mmap()'ing (with MAP_FIXED) a gntdev mapping over the
same address range as a previously set up gntdev mapping. This scenario
can be summarized with the following call-trace, which was valid prior
to this commit:
mmap
gntdev_mmap
mmap (repeat mmap with MAP_FIXED over the same address range)
gntdev_invalidate
unmap_grant_pages (sets 'being_removed' entries to true)
gnttab_unmap_refs_async
unmap_single_vma
gntdev_mmap (maps the shared pages again)
munmap
gntdev_invalidate
unmap_grant_pages
(no-op because 'being_removed' entries are true)
unmap_single_vma (For PV domains, Xen reports that a granted page
is being unmapped and triggers a general protection fault in the
affected domain, if Xen was built with CONFIG_DEBUG)
The fix for this last scenario could be worth its own commit, but we
opted for a single commit, because removing the gntdev_grant_map
structure's vma field requires guarding the entry to gntdev_mmap(), and
the live_grants atomic counter is not sufficient on its own to prevent
the mmap() over a pre-existing mapping.
Link: https://github.com/QubesOS/qubes-issues/issues/7631
Fixes: ab31523c2fca ("xen/gntdev: allow usermode to map granted pages")
Cc: stable@vger.kernel.org
Signed-off-by: M. Vefa Bicakci <m.v.b@runbox.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/r/20221002222006.2077-3-m.v.b@runbox.com
Signed-off-by: Juergen Gross <jgross@suse.com>
2022-10-02 15:20:06 -07:00
|
|
|
atomic_t in_use;
|
2019-11-12 13:22:31 -07:00
|
|
|
struct mmu_interval_notifier notifier;
|
xen/gntdev: Accommodate VMA splitting
Prior to this commit, the gntdev driver code did not handle the
following scenario correctly with paravirtualized (PV) Xen domains:
* User process sets up a gntdev mapping composed of two grant mappings
(i.e., two pages shared by another Xen domain).
* User process munmap()s one of the pages.
* User process munmap()s the remaining page.
* User process exits.
In the scenario above, the user process would cause the kernel to log
the following messages in dmesg for the first munmap(), and the second
munmap() call would result in similar log messages:
BUG: Bad page map in process doublemap.test pte:... pmd:...
page:0000000057c97bff refcount:1 mapcount:-1 \
mapping:0000000000000000 index:0x0 pfn:...
...
page dumped because: bad pte
...
file:gntdev fault:0x0 mmap:gntdev_mmap [xen_gntdev] readpage:0x0
...
Call Trace:
<TASK>
dump_stack_lvl+0x46/0x5e
print_bad_pte.cold+0x66/0xb6
unmap_page_range+0x7e5/0xdc0
unmap_vmas+0x78/0xf0
unmap_region+0xa8/0x110
__do_munmap+0x1ea/0x4e0
__vm_munmap+0x75/0x120
__x64_sys_munmap+0x28/0x40
do_syscall_64+0x38/0x90
entry_SYSCALL_64_after_hwframe+0x61/0xcb
...
For each munmap() call, the Xen hypervisor (if built with CONFIG_DEBUG)
would print out the following and trigger a general protection fault in
the affected Xen PV domain:
(XEN) d0v... Attempt to implicitly unmap d0's grant PTE ...
(XEN) d0v... Attempt to implicitly unmap d0's grant PTE ...
As of this writing, gntdev_grant_map structure's vma field (referred to
as map->vma below) is mainly used for checking the start and end
addresses of mappings. However, with split VMAs, these may change, and
there could be more than one VMA associated with a gntdev mapping.
Hence, remove the use of map->vma and rely on map->pages_vm_start for
the original start address and on (map->count << PAGE_SHIFT) for the
original mapping size. Let the invalidate() and find_special_page()
hooks use these.
Also, given that there can be multiple VMAs associated with a gntdev
mapping, move the "mmu_interval_notifier_remove(&map->notifier)" call to
the end of gntdev_put_map, so that the MMU notifier is only removed
after the closing of the last remaining VMA.
Finally, use an atomic to prevent inadvertent gntdev mapping re-use,
instead of using the map->live_grants atomic counter and/or the map->vma
pointer (the latter of which is now removed). This prevents the
userspace from mmap()'ing (with MAP_FIXED) a gntdev mapping over the
same address range as a previously set up gntdev mapping. This scenario
can be summarized with the following call-trace, which was valid prior
to this commit:
mmap
gntdev_mmap
mmap (repeat mmap with MAP_FIXED over the same address range)
gntdev_invalidate
unmap_grant_pages (sets 'being_removed' entries to true)
gnttab_unmap_refs_async
unmap_single_vma
gntdev_mmap (maps the shared pages again)
munmap
gntdev_invalidate
unmap_grant_pages
(no-op because 'being_removed' entries are true)
unmap_single_vma (For PV domains, Xen reports that a granted page
is being unmapped and triggers a general protection fault in the
affected domain, if Xen was built with CONFIG_DEBUG)
The fix for this last scenario could be worth its own commit, but we
opted for a single commit, because removing the gntdev_grant_map
structure's vma field requires guarding the entry to gntdev_mmap(), and
the live_grants atomic counter is not sufficient on its own to prevent
the mmap() over a pre-existing mapping.
Link: https://github.com/QubesOS/qubes-issues/issues/7631
Fixes: ab31523c2fca ("xen/gntdev: allow usermode to map granted pages")
Cc: stable@vger.kernel.org
Signed-off-by: M. Vefa Bicakci <m.v.b@runbox.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/r/20221002222006.2077-3-m.v.b@runbox.com
Signed-off-by: Juergen Gross <jgross@suse.com>
2022-10-02 15:20:06 -07:00
|
|
|
bool notifier_init;
|
2018-07-20 02:01:47 -07:00
|
|
|
struct list_head next;
|
|
|
|
|
int index;
|
|
|
|
|
int count;
|
|
|
|
|
int flags;
|
|
|
|
|
refcount_t users;
|
|
|
|
|
struct gntdev_unmap_notify notify;
|
|
|
|
|
struct ioctl_gntdev_grant_ref *grants;
|
|
|
|
|
struct gnttab_map_grant_ref *map_ops;
|
|
|
|
|
struct gnttab_unmap_grant_ref *unmap_ops;
|
|
|
|
|
struct gnttab_map_grant_ref *kmap_ops;
|
|
|
|
|
struct gnttab_unmap_grant_ref *kunmap_ops;
|
xen/gntdev: Avoid blocking in unmap_grant_pages()
unmap_grant_pages() currently waits for the pages to no longer be used.
In https://github.com/QubesOS/qubes-issues/issues/7481, this lead to a
deadlock against i915: i915 was waiting for gntdev's MMU notifier to
finish, while gntdev was waiting for i915 to free its pages. I also
believe this is responsible for various deadlocks I have experienced in
the past.
Avoid these problems by making unmap_grant_pages async. This requires
making it return void, as any errors will not be available when the
function returns. Fortunately, the only use of the return value is a
WARN_ON(), which can be replaced by a WARN_ON when the error is
detected. Additionally, a failed call will not prevent further calls
from being made, but this is harmless.
Because unmap_grant_pages is now async, the grant handle will be sent to
INVALID_GRANT_HANDLE too late to prevent multiple unmaps of the same
handle. Instead, a separate bool array is allocated for this purpose.
This wastes memory, but stuffing this information in padding bytes is
too fragile. Furthermore, it is necessary to grab a reference to the
map before making the asynchronous call, and release the reference when
the call returns.
It is also necessary to guard against reentrancy in gntdev_map_put(),
and to handle the case where userspace tries to map a mapping whose
contents have not all been freed yet.
Fixes: 745282256c75 ("xen/gntdev: safely unmap grants in case they are still in use")
Cc: stable@vger.kernel.org
Signed-off-by: Demi Marie Obenour <demi@invisiblethingslab.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/r/20220622022726.2538-1-demi@invisiblethingslab.com
Signed-off-by: Juergen Gross <jgross@suse.com>
2022-06-21 19:27:26 -07:00
|
|
|
bool *being_removed;
|
2018-07-20 02:01:47 -07:00
|
|
|
struct page **pages;
|
|
|
|
|
unsigned long pages_vm_start;
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_XEN_GRANT_DMA_ALLOC
|
|
|
|
|
/*
|
|
|
|
|
* If dmabuf_vaddr is not NULL then this mapping is backed by DMA
|
|
|
|
|
* capable memory.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
struct device *dma_dev;
|
|
|
|
|
/* Flags used to create this DMA buffer: GNTDEV_DMA_FLAG_XXX. */
|
|
|
|
|
int dma_flags;
|
|
|
|
|
void *dma_vaddr;
|
|
|
|
|
dma_addr_t dma_bus_addr;
|
|
|
|
|
/* Needed to avoid allocation in gnttab_dma_free_pages(). */
|
|
|
|
|
xen_pfn_t *frames;
|
|
|
|
|
#endif
|
xen/gntdev: Avoid blocking in unmap_grant_pages()
unmap_grant_pages() currently waits for the pages to no longer be used.
In https://github.com/QubesOS/qubes-issues/issues/7481, this lead to a
deadlock against i915: i915 was waiting for gntdev's MMU notifier to
finish, while gntdev was waiting for i915 to free its pages. I also
believe this is responsible for various deadlocks I have experienced in
the past.
Avoid these problems by making unmap_grant_pages async. This requires
making it return void, as any errors will not be available when the
function returns. Fortunately, the only use of the return value is a
WARN_ON(), which can be replaced by a WARN_ON when the error is
detected. Additionally, a failed call will not prevent further calls
from being made, but this is harmless.
Because unmap_grant_pages is now async, the grant handle will be sent to
INVALID_GRANT_HANDLE too late to prevent multiple unmaps of the same
handle. Instead, a separate bool array is allocated for this purpose.
This wastes memory, but stuffing this information in padding bytes is
too fragile. Furthermore, it is necessary to grab a reference to the
map before making the asynchronous call, and release the reference when
the call returns.
It is also necessary to guard against reentrancy in gntdev_map_put(),
and to handle the case where userspace tries to map a mapping whose
contents have not all been freed yet.
Fixes: 745282256c75 ("xen/gntdev: safely unmap grants in case they are still in use")
Cc: stable@vger.kernel.org
Signed-off-by: Demi Marie Obenour <demi@invisiblethingslab.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/r/20220622022726.2538-1-demi@invisiblethingslab.com
Signed-off-by: Juergen Gross <jgross@suse.com>
2022-06-21 19:27:26 -07:00
|
|
|
|
|
|
|
|
/* Number of live grants */
|
|
|
|
|
atomic_t live_grants;
|
|
|
|
|
/* Needed to avoid allocation in __unmap_grant_pages */
|
|
|
|
|
struct gntab_unmap_queue_data unmap_data;
|
2018-07-20 02:01:47 -07:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct gntdev_grant_map *gntdev_alloc_map(struct gntdev_priv *priv, int count,
|
|
|
|
|
int dma_flags);
|
|
|
|
|
|
|
|
|
|
void gntdev_add_map(struct gntdev_priv *priv, struct gntdev_grant_map *add);
|
|
|
|
|
|
|
|
|
|
void gntdev_put_map(struct gntdev_priv *priv, struct gntdev_grant_map *map);
|
|
|
|
|
|
2019-11-07 04:15:45 -07:00
|
|
|
bool gntdev_test_page_count(unsigned int count);
|
2018-07-20 02:01:47 -07:00
|
|
|
|
|
|
|
|
int gntdev_map_grant_pages(struct gntdev_grant_map *map);
|
|
|
|
|
|
|
|
|
|
#endif
|