mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
synced 2025-04-19 20:58:31 +09:00
ring-buffer updates for v6.15
- Restructure the persistent memory to have a "scratch" area Instead of hard coding the KASLR offset in the persistent memory by the ring buffer, push that work up to the callers of the persistent memory as they are the ones that need this information. The offsets and such is not important to the ring buffer logic and it should not be part of that. A scratch pad is now created when the caller allocates a ring buffer from persistent memory by stating how much memory it needs to save. - Allow where modules are loaded to be saved in the new scratch pad Save the addresses of modules when they are loaded into the persistent memory scratch pad. - A new module_for_each_mod() helper function was created With the acknowledgement of the module maintainers a new module helper function was created to iterate over all the currently loaded modules. This has a callback to be called for each module. This is needed for when tracing is started in the persistent buffer and the currently loaded modules need to be saved in the scratch area. - Expose the last boot information where the kernel and modules were loaded The last_boot_info file is updated to print out the addresses of where the kernel "_text" location was loaded from a previous boot, as well as where the modules are loaded. If the buffer is recording the current boot, it only prints "# Current" so that it does not expose the KASLR offset of the currently running kernel. - Allow the persistent ring buffer to be released (freed) To have this in production environments, where the kernel command line can not be changed easily, the ring buffer needs to be freed when it is not going to be used. The memory for the buffer will always be allocated at boot up, but if the system isn't going to enable tracing, the memory needs to be freed. Allow it to be freed and added back to the kernel memory pool. - Allow stack traces to print the function names in the persistent buffer Now that the modules are saved in the persistent ring buffer, if the same modules are loaded, the printing of the function names will examine the saved modules. If the module is found in the scratch area and is also loaded, then it will do the offset shift and use kallsyms to display the function name. If the address is not found, it simply displays the address from the previous boot in hex. -----BEGIN PGP SIGNATURE----- iIoEABYIADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCZ+cUERQccm9zdGVkdEBn b29kbWlzLm9yZwAKCRAp5XQQmuv6qrAsAQCFt2nfzxoe3wtF5EqIT1VHp/8bQVjG gBe8B6ouboreogD/dS7yK8MRy24ZAmObGwYG0RbVicd50S7P8Rf7+823ng8= =OJKk -----END PGP SIGNATURE----- Merge tag 'trace-ringbuffer-v6.15-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull ring-buffer updates from Steven Rostedt: - Restructure the persistent memory to have a "scratch" area Instead of hard coding the KASLR offset in the persistent memory by the ring buffer, push that work up to the callers of the persistent memory as they are the ones that need this information. The offsets and such is not important to the ring buffer logic and it should not be part of that. A scratch pad is now created when the caller allocates a ring buffer from persistent memory by stating how much memory it needs to save. - Allow where modules are loaded to be saved in the new scratch pad Save the addresses of modules when they are loaded into the persistent memory scratch pad. - A new module_for_each_mod() helper function was created With the acknowledgement of the module maintainers a new module helper function was created to iterate over all the currently loaded modules. This has a callback to be called for each module. This is needed for when tracing is started in the persistent buffer and the currently loaded modules need to be saved in the scratch area. - Expose the last boot information where the kernel and modules were loaded The last_boot_info file is updated to print out the addresses of where the kernel "_text" location was loaded from a previous boot, as well as where the modules are loaded. If the buffer is recording the current boot, it only prints "# Current" so that it does not expose the KASLR offset of the currently running kernel. - Allow the persistent ring buffer to be released (freed) To have this in production environments, where the kernel command line can not be changed easily, the ring buffer needs to be freed when it is not going to be used. The memory for the buffer will always be allocated at boot up, but if the system isn't going to enable tracing, the memory needs to be freed. Allow it to be freed and added back to the kernel memory pool. - Allow stack traces to print the function names in the persistent buffer Now that the modules are saved in the persistent ring buffer, if the same modules are loaded, the printing of the function names will examine the saved modules. If the module is found in the scratch area and is also loaded, then it will do the offset shift and use kallsyms to display the function name. If the address is not found, it simply displays the address from the previous boot in hex. * tag 'trace-ringbuffer-v6.15-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: tracing: Use _text and the kernel offset in last_boot_info tracing: Show last module text symbols in the stacktrace ring-buffer: Remove the unused variable bmeta tracing: Skip update_last_data() if cleared and remove active check for save_mod() tracing: Initialize scratch_size to zero to prevent UB tracing: Fix a compilation error without CONFIG_MODULES tracing: Freeable reserved ring buffer mm/memblock: Add reserved memory release function tracing: Update modules to persistent instances when loaded tracing: Show module names and addresses of last boot tracing: Have persistent trace instances save module addresses module: Add module_for_each_mod() function tracing: Have persistent trace instances save KASLR offset ring-buffer: Add ring_buffer_meta_scratch() ring-buffer: Add buffer meta data for persistent ring buffer ring-buffer: Use kaslr address instead of text delta ring-buffer: Fix bytes_dropped calculation issue
This commit is contained in:
Linus Torvalds
commit
46d29f23a7
@ -4110,6 +4110,7 @@ void vma_pgtable_walk_begin(struct vm_area_struct *vma);
|
||||
void vma_pgtable_walk_end(struct vm_area_struct *vma);
|
||||
|
||||
int reserve_mem_find_by_name(const char *name, phys_addr_t *start, phys_addr_t *size);
|
||||
int reserve_mem_release_by_name(const char *name);
|
||||
|
||||
#ifdef CONFIG_64BIT
|
||||
int do_mseal(unsigned long start, size_t len_in, unsigned long flags);
|
||||
|
||||
@ -771,6 +771,8 @@ static inline bool is_livepatch_module(struct module *mod)
|
||||
|
||||
void set_module_sig_enforced(void);
|
||||
|
||||
void module_for_each_mod(int(*func)(struct module *mod, void *data), void *data);
|
||||
|
||||
#else /* !CONFIG_MODULES... */
|
||||
|
||||
static inline struct module *__module_address(unsigned long addr)
|
||||
@ -878,6 +880,10 @@ static inline bool module_is_coming(struct module *mod)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
static inline void module_for_each_mod(int(*func)(struct module *mod, void *data), void *data)
|
||||
{
|
||||
}
|
||||
#endif /* CONFIG_MODULES */
|
||||
|
||||
#ifdef CONFIG_SYSFS
|
||||
|
||||
@ -92,10 +92,10 @@ __ring_buffer_alloc(unsigned long size, unsigned flags, struct lock_class_key *k
|
||||
struct trace_buffer *__ring_buffer_alloc_range(unsigned long size, unsigned flags,
|
||||
int order, unsigned long start,
|
||||
unsigned long range_size,
|
||||
unsigned long scratch_size,
|
||||
struct lock_class_key *key);
|
||||
|
||||
bool ring_buffer_last_boot_delta(struct trace_buffer *buffer, long *text,
|
||||
long *data);
|
||||
void *ring_buffer_meta_scratch(struct trace_buffer *buffer, unsigned int *size);
|
||||
|
||||
/*
|
||||
* Because the ring buffer is generic, if other users of the ring buffer get
|
||||
@ -113,11 +113,11 @@ bool ring_buffer_last_boot_delta(struct trace_buffer *buffer, long *text,
|
||||
* traced by ftrace, it can produce lockdep warnings. We need to keep each
|
||||
* ring buffer's lock class separate.
|
||||
*/
|
||||
#define ring_buffer_alloc_range(size, flags, order, start, range_size) \
|
||||
#define ring_buffer_alloc_range(size, flags, order, start, range_size, s_size) \
|
||||
({ \
|
||||
static struct lock_class_key __key; \
|
||||
__ring_buffer_alloc_range((size), (flags), (order), (start), \
|
||||
(range_size), &__key); \
|
||||
(range_size), (s_size), &__key); \
|
||||
})
|
||||
|
||||
typedef bool (*ring_buffer_cond_fn)(void *data);
|
||||
|
||||
@ -3744,6 +3744,19 @@ bool is_module_text_address(unsigned long addr)
|
||||
return __module_text_address(addr) != NULL;
|
||||
}
|
||||
|
||||
void module_for_each_mod(int(*func)(struct module *mod, void *data), void *data)
|
||||
{
|
||||
struct module *mod;
|
||||
|
||||
guard(rcu)();
|
||||
list_for_each_entry_rcu(mod, &modules, list) {
|
||||
if (mod->state == MODULE_STATE_UNFORMED)
|
||||
continue;
|
||||
if (func(mod, data))
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* __module_text_address() - get the module whose code contains an address.
|
||||
* @addr: the address.
|
||||
|
||||
@ -31,6 +31,7 @@
|
||||
|
||||
#include <asm/local64.h>
|
||||
#include <asm/local.h>
|
||||
#include <asm/setup.h>
|
||||
|
||||
#include "trace.h"
|
||||
|
||||
@ -48,9 +49,12 @@ static void update_pages_handler(struct work_struct *work);
|
||||
|
||||
struct ring_buffer_meta {
|
||||
int magic;
|
||||
int struct_size;
|
||||
unsigned long text_addr;
|
||||
unsigned long data_addr;
|
||||
int struct_sizes;
|
||||
unsigned long total_size;
|
||||
unsigned long buffers_offset;
|
||||
};
|
||||
|
||||
struct ring_buffer_cpu_meta {
|
||||
unsigned long first_buffer;
|
||||
unsigned long head_buffer;
|
||||
unsigned long commit_buffer;
|
||||
@ -517,7 +521,7 @@ struct ring_buffer_per_cpu {
|
||||
struct mutex mapping_lock;
|
||||
unsigned long *subbuf_ids; /* ID to subbuf VA */
|
||||
struct trace_buffer_meta *meta_page;
|
||||
struct ring_buffer_meta *ring_meta;
|
||||
struct ring_buffer_cpu_meta *ring_meta;
|
||||
|
||||
/* ring buffer pages to update, > 0 to add, < 0 to remove */
|
||||
long nr_pages_to_update;
|
||||
@ -550,8 +554,7 @@ struct trace_buffer {
|
||||
unsigned long range_addr_start;
|
||||
unsigned long range_addr_end;
|
||||
|
||||
long last_text_delta;
|
||||
long last_data_delta;
|
||||
struct ring_buffer_meta *meta;
|
||||
|
||||
unsigned int subbuf_size;
|
||||
unsigned int subbuf_order;
|
||||
@ -1271,7 +1274,7 @@ static void rb_head_page_activate(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
rb_set_list_to_head(head->list.prev);
|
||||
|
||||
if (cpu_buffer->ring_meta) {
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
meta->head_buffer = (unsigned long)head->page;
|
||||
}
|
||||
}
|
||||
@ -1569,7 +1572,7 @@ out_locked:
|
||||
static unsigned long
|
||||
rb_range_align_subbuf(unsigned long addr, int subbuf_size, int nr_subbufs)
|
||||
{
|
||||
addr += sizeof(struct ring_buffer_meta) +
|
||||
addr += sizeof(struct ring_buffer_cpu_meta) +
|
||||
sizeof(int) * nr_subbufs;
|
||||
return ALIGN(addr, subbuf_size);
|
||||
}
|
||||
@ -1580,19 +1583,22 @@ rb_range_align_subbuf(unsigned long addr, int subbuf_size, int nr_subbufs)
|
||||
static void *rb_range_meta(struct trace_buffer *buffer, int nr_pages, int cpu)
|
||||
{
|
||||
int subbuf_size = buffer->subbuf_size + BUF_PAGE_HDR_SIZE;
|
||||
unsigned long ptr = buffer->range_addr_start;
|
||||
struct ring_buffer_meta *meta;
|
||||
struct ring_buffer_cpu_meta *meta;
|
||||
struct ring_buffer_meta *bmeta;
|
||||
unsigned long ptr;
|
||||
int nr_subbufs;
|
||||
|
||||
if (!ptr)
|
||||
bmeta = buffer->meta;
|
||||
if (!bmeta)
|
||||
return NULL;
|
||||
|
||||
ptr = (unsigned long)bmeta + bmeta->buffers_offset;
|
||||
meta = (struct ring_buffer_cpu_meta *)ptr;
|
||||
|
||||
/* When nr_pages passed in is zero, the first meta has already been initialized */
|
||||
if (!nr_pages) {
|
||||
meta = (struct ring_buffer_meta *)ptr;
|
||||
nr_subbufs = meta->nr_subbufs;
|
||||
} else {
|
||||
meta = NULL;
|
||||
/* Include the reader page */
|
||||
nr_subbufs = nr_pages + 1;
|
||||
}
|
||||
@ -1624,7 +1630,7 @@ static void *rb_range_meta(struct trace_buffer *buffer, int nr_pages, int cpu)
|
||||
}
|
||||
|
||||
/* Return the start of subbufs given the meta pointer */
|
||||
static void *rb_subbufs_from_meta(struct ring_buffer_meta *meta)
|
||||
static void *rb_subbufs_from_meta(struct ring_buffer_cpu_meta *meta)
|
||||
{
|
||||
int subbuf_size = meta->subbuf_size;
|
||||
unsigned long ptr;
|
||||
@ -1640,7 +1646,7 @@ static void *rb_subbufs_from_meta(struct ring_buffer_meta *meta)
|
||||
*/
|
||||
static void *rb_range_buffer(struct ring_buffer_per_cpu *cpu_buffer, int idx)
|
||||
{
|
||||
struct ring_buffer_meta *meta;
|
||||
struct ring_buffer_cpu_meta *meta;
|
||||
unsigned long ptr;
|
||||
int subbuf_size;
|
||||
|
||||
@ -1665,15 +1671,78 @@ static void *rb_range_buffer(struct ring_buffer_per_cpu *cpu_buffer, int idx)
|
||||
return (void *)ptr;
|
||||
}
|
||||
|
||||
/*
|
||||
* See if the existing memory contains a valid meta section.
|
||||
* if so, use that, otherwise initialize it.
|
||||
*/
|
||||
static bool rb_meta_init(struct trace_buffer *buffer, int scratch_size)
|
||||
{
|
||||
unsigned long ptr = buffer->range_addr_start;
|
||||
struct ring_buffer_meta *bmeta;
|
||||
unsigned long total_size;
|
||||
int struct_sizes;
|
||||
|
||||
bmeta = (struct ring_buffer_meta *)ptr;
|
||||
buffer->meta = bmeta;
|
||||
|
||||
total_size = buffer->range_addr_end - buffer->range_addr_start;
|
||||
|
||||
struct_sizes = sizeof(struct ring_buffer_cpu_meta);
|
||||
struct_sizes |= sizeof(*bmeta) << 16;
|
||||
|
||||
/* The first buffer will start word size after the meta page */
|
||||
ptr += sizeof(*bmeta);
|
||||
ptr = ALIGN(ptr, sizeof(long));
|
||||
ptr += scratch_size;
|
||||
|
||||
if (bmeta->magic != RING_BUFFER_META_MAGIC) {
|
||||
pr_info("Ring buffer boot meta mismatch of magic\n");
|
||||
goto init;
|
||||
}
|
||||
|
||||
if (bmeta->struct_sizes != struct_sizes) {
|
||||
pr_info("Ring buffer boot meta mismatch of struct size\n");
|
||||
goto init;
|
||||
}
|
||||
|
||||
if (bmeta->total_size != total_size) {
|
||||
pr_info("Ring buffer boot meta mismatch of total size\n");
|
||||
goto init;
|
||||
}
|
||||
|
||||
if (bmeta->buffers_offset > bmeta->total_size) {
|
||||
pr_info("Ring buffer boot meta mismatch of offset outside of total size\n");
|
||||
goto init;
|
||||
}
|
||||
|
||||
if (bmeta->buffers_offset != (void *)ptr - (void *)bmeta) {
|
||||
pr_info("Ring buffer boot meta mismatch of first buffer offset\n");
|
||||
goto init;
|
||||
}
|
||||
|
||||
return true;
|
||||
|
||||
init:
|
||||
bmeta->magic = RING_BUFFER_META_MAGIC;
|
||||
bmeta->struct_sizes = struct_sizes;
|
||||
bmeta->total_size = total_size;
|
||||
bmeta->buffers_offset = (void *)ptr - (void *)bmeta;
|
||||
|
||||
/* Zero out the scatch pad */
|
||||
memset((void *)bmeta + sizeof(*bmeta), 0, bmeta->buffers_offset - sizeof(*bmeta));
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/*
|
||||
* See if the existing memory contains valid ring buffer data.
|
||||
* As the previous kernel must be the same as this kernel, all
|
||||
* the calculations (size of buffers and number of buffers)
|
||||
* must be the same.
|
||||
*/
|
||||
static bool rb_meta_valid(struct ring_buffer_meta *meta, int cpu,
|
||||
struct trace_buffer *buffer, int nr_pages,
|
||||
unsigned long *subbuf_mask)
|
||||
static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
|
||||
struct trace_buffer *buffer, int nr_pages,
|
||||
unsigned long *subbuf_mask)
|
||||
{
|
||||
int subbuf_size = PAGE_SIZE;
|
||||
struct buffer_data_page *subbuf;
|
||||
@ -1684,20 +1753,6 @@ static bool rb_meta_valid(struct ring_buffer_meta *meta, int cpu,
|
||||
if (!subbuf_mask)
|
||||
return false;
|
||||
|
||||
/* Check the meta magic and meta struct size */
|
||||
if (meta->magic != RING_BUFFER_META_MAGIC ||
|
||||
meta->struct_size != sizeof(*meta)) {
|
||||
pr_info("Ring buffer boot meta[%d] mismatch of magic or struct size\n", cpu);
|
||||
return false;
|
||||
}
|
||||
|
||||
/* The subbuffer's size and number of subbuffers must match */
|
||||
if (meta->subbuf_size != subbuf_size ||
|
||||
meta->nr_subbufs != nr_pages + 1) {
|
||||
pr_info("Ring buffer boot meta [%d] mismatch of subbuf_size/nr_pages\n", cpu);
|
||||
return false;
|
||||
}
|
||||
|
||||
buffers_start = meta->first_buffer;
|
||||
buffers_end = meta->first_buffer + (subbuf_size * meta->nr_subbufs);
|
||||
|
||||
@ -1743,7 +1798,7 @@ static bool rb_meta_valid(struct ring_buffer_meta *meta, int cpu,
|
||||
return true;
|
||||
}
|
||||
|
||||
static int rb_meta_subbuf_idx(struct ring_buffer_meta *meta, void *subbuf);
|
||||
static int rb_meta_subbuf_idx(struct ring_buffer_cpu_meta *meta, void *subbuf);
|
||||
|
||||
static int rb_read_data_buffer(struct buffer_data_page *dpage, int tail, int cpu,
|
||||
unsigned long long *timestamp, u64 *delta_ptr)
|
||||
@ -1810,7 +1865,7 @@ static int rb_validate_buffer(struct buffer_data_page *dpage, int cpu)
|
||||
/* If the meta data has been validated, now validate the events */
|
||||
static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
{
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
struct buffer_page *head_page;
|
||||
unsigned long entry_bytes = 0;
|
||||
unsigned long entries = 0;
|
||||
@ -1891,24 +1946,13 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
}
|
||||
}
|
||||
|
||||
/* Used to calculate data delta */
|
||||
static char rb_data_ptr[] = "";
|
||||
|
||||
#define THIS_TEXT_PTR ((unsigned long)rb_meta_init_text_addr)
|
||||
#define THIS_DATA_PTR ((unsigned long)rb_data_ptr)
|
||||
|
||||
static void rb_meta_init_text_addr(struct ring_buffer_meta *meta)
|
||||
static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages, int scratch_size)
|
||||
{
|
||||
meta->text_addr = THIS_TEXT_PTR;
|
||||
meta->data_addr = THIS_DATA_PTR;
|
||||
}
|
||||
|
||||
static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages)
|
||||
{
|
||||
struct ring_buffer_meta *meta;
|
||||
struct ring_buffer_cpu_meta *meta;
|
||||
unsigned long *subbuf_mask;
|
||||
unsigned long delta;
|
||||
void *subbuf;
|
||||
bool valid = false;
|
||||
int cpu;
|
||||
int i;
|
||||
|
||||
@ -1916,20 +1960,21 @@ static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages)
|
||||
subbuf_mask = bitmap_alloc(nr_pages + 1, GFP_KERNEL);
|
||||
/* If subbuf_mask fails to allocate, then rb_meta_valid() will return false */
|
||||
|
||||
if (rb_meta_init(buffer, scratch_size))
|
||||
valid = true;
|
||||
|
||||
for (cpu = 0; cpu < nr_cpu_ids; cpu++) {
|
||||
void *next_meta;
|
||||
|
||||
meta = rb_range_meta(buffer, nr_pages, cpu);
|
||||
|
||||
if (rb_meta_valid(meta, cpu, buffer, nr_pages, subbuf_mask)) {
|
||||
if (valid && rb_cpu_meta_valid(meta, cpu, buffer, nr_pages, subbuf_mask)) {
|
||||
/* Make the mappings match the current address */
|
||||
subbuf = rb_subbufs_from_meta(meta);
|
||||
delta = (unsigned long)subbuf - meta->first_buffer;
|
||||
meta->first_buffer += delta;
|
||||
meta->head_buffer += delta;
|
||||
meta->commit_buffer += delta;
|
||||
buffer->last_text_delta = THIS_TEXT_PTR - meta->text_addr;
|
||||
buffer->last_data_delta = THIS_DATA_PTR - meta->data_addr;
|
||||
continue;
|
||||
}
|
||||
|
||||
@ -1940,16 +1985,12 @@ static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages)
|
||||
|
||||
memset(meta, 0, next_meta - (void *)meta);
|
||||
|
||||
meta->magic = RING_BUFFER_META_MAGIC;
|
||||
meta->struct_size = sizeof(*meta);
|
||||
|
||||
meta->nr_subbufs = nr_pages + 1;
|
||||
meta->subbuf_size = PAGE_SIZE;
|
||||
|
||||
subbuf = rb_subbufs_from_meta(meta);
|
||||
|
||||
meta->first_buffer = (unsigned long)subbuf;
|
||||
rb_meta_init_text_addr(meta);
|
||||
|
||||
/*
|
||||
* The buffers[] array holds the order of the sub-buffers
|
||||
@ -1971,7 +2012,7 @@ static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages)
|
||||
static void *rbm_start(struct seq_file *m, loff_t *pos)
|
||||
{
|
||||
struct ring_buffer_per_cpu *cpu_buffer = m->private;
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
unsigned long val;
|
||||
|
||||
if (!meta)
|
||||
@ -1996,7 +2037,7 @@ static void *rbm_next(struct seq_file *m, void *v, loff_t *pos)
|
||||
static int rbm_show(struct seq_file *m, void *v)
|
||||
{
|
||||
struct ring_buffer_per_cpu *cpu_buffer = m->private;
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
unsigned long val = (unsigned long)v;
|
||||
|
||||
if (val == 1) {
|
||||
@ -2045,7 +2086,7 @@ int ring_buffer_meta_seq_init(struct file *file, struct trace_buffer *buffer, in
|
||||
static void rb_meta_buffer_update(struct ring_buffer_per_cpu *cpu_buffer,
|
||||
struct buffer_page *bpage)
|
||||
{
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
|
||||
if (meta->head_buffer == (unsigned long)bpage->page)
|
||||
cpu_buffer->head_page = bpage;
|
||||
@ -2060,7 +2101,7 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
|
||||
long nr_pages, struct list_head *pages)
|
||||
{
|
||||
struct trace_buffer *buffer = cpu_buffer->buffer;
|
||||
struct ring_buffer_meta *meta = NULL;
|
||||
struct ring_buffer_cpu_meta *meta = NULL;
|
||||
struct buffer_page *bpage, *tmp;
|
||||
bool user_thread = current->mm != NULL;
|
||||
gfp_t mflags;
|
||||
@ -2184,7 +2225,7 @@ static struct ring_buffer_per_cpu *
|
||||
rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)
|
||||
{
|
||||
struct ring_buffer_per_cpu *cpu_buffer;
|
||||
struct ring_buffer_meta *meta;
|
||||
struct ring_buffer_cpu_meta *meta;
|
||||
struct buffer_page *bpage;
|
||||
struct page *page;
|
||||
int ret;
|
||||
@ -2313,6 +2354,7 @@ static void rb_free_cpu_buffer(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
|
||||
int order, unsigned long start,
|
||||
unsigned long end,
|
||||
unsigned long scratch_size,
|
||||
struct lock_class_key *key)
|
||||
{
|
||||
struct trace_buffer *buffer;
|
||||
@ -2355,10 +2397,23 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
|
||||
|
||||
/* If start/end are specified, then that overrides size */
|
||||
if (start && end) {
|
||||
unsigned long buffers_start;
|
||||
unsigned long ptr;
|
||||
int n;
|
||||
|
||||
size = end - start;
|
||||
/* Make sure that start is word aligned */
|
||||
start = ALIGN(start, sizeof(long));
|
||||
|
||||
/* scratch_size needs to be aligned too */
|
||||
scratch_size = ALIGN(scratch_size, sizeof(long));
|
||||
|
||||
/* Subtract the buffer meta data and word aligned */
|
||||
buffers_start = start + sizeof(struct ring_buffer_cpu_meta);
|
||||
buffers_start = ALIGN(buffers_start, sizeof(long));
|
||||
buffers_start += scratch_size;
|
||||
|
||||
/* Calculate the size for the per CPU data */
|
||||
size = end - buffers_start;
|
||||
size = size / nr_cpu_ids;
|
||||
|
||||
/*
|
||||
@ -2368,7 +2423,7 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
|
||||
* needed, plus account for the integer array index that
|
||||
* will be appended to the meta data.
|
||||
*/
|
||||
nr_pages = (size - sizeof(struct ring_buffer_meta)) /
|
||||
nr_pages = (size - sizeof(struct ring_buffer_cpu_meta)) /
|
||||
(subbuf_size + sizeof(int));
|
||||
/* Need at least two pages plus the reader page */
|
||||
if (nr_pages < 3)
|
||||
@ -2376,8 +2431,8 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
|
||||
|
||||
again:
|
||||
/* Make sure that the size fits aligned */
|
||||
for (n = 0, ptr = start; n < nr_cpu_ids; n++) {
|
||||
ptr += sizeof(struct ring_buffer_meta) +
|
||||
for (n = 0, ptr = buffers_start; n < nr_cpu_ids; n++) {
|
||||
ptr += sizeof(struct ring_buffer_cpu_meta) +
|
||||
sizeof(int) * nr_pages;
|
||||
ptr = ALIGN(ptr, subbuf_size);
|
||||
ptr += subbuf_size * nr_pages;
|
||||
@ -2394,7 +2449,7 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
|
||||
buffer->range_addr_start = start;
|
||||
buffer->range_addr_end = end;
|
||||
|
||||
rb_range_meta_init(buffer, nr_pages);
|
||||
rb_range_meta_init(buffer, nr_pages, scratch_size);
|
||||
} else {
|
||||
|
||||
/* need at least two pages */
|
||||
@ -2447,7 +2502,7 @@ struct trace_buffer *__ring_buffer_alloc(unsigned long size, unsigned flags,
|
||||
struct lock_class_key *key)
|
||||
{
|
||||
/* Default buffer page size - one system page */
|
||||
return alloc_buffer(size, flags, 0, 0, 0,key);
|
||||
return alloc_buffer(size, flags, 0, 0, 0, 0, key);
|
||||
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(__ring_buffer_alloc);
|
||||
@ -2459,6 +2514,7 @@ EXPORT_SYMBOL_GPL(__ring_buffer_alloc);
|
||||
* @order: sub-buffer order
|
||||
* @start: start of allocated range
|
||||
* @range_size: size of allocated range
|
||||
* @scratch_size: size of scratch area (for preallocated memory buffers)
|
||||
* @key: ring buffer reader_lock_key.
|
||||
*
|
||||
* Currently the only flag that is available is the RB_FL_OVERWRITE
|
||||
@ -2469,32 +2525,29 @@ EXPORT_SYMBOL_GPL(__ring_buffer_alloc);
|
||||
struct trace_buffer *__ring_buffer_alloc_range(unsigned long size, unsigned flags,
|
||||
int order, unsigned long start,
|
||||
unsigned long range_size,
|
||||
unsigned long scratch_size,
|
||||
struct lock_class_key *key)
|
||||
{
|
||||
return alloc_buffer(size, flags, order, start, start + range_size, key);
|
||||
return alloc_buffer(size, flags, order, start, start + range_size,
|
||||
scratch_size, key);
|
||||
}
|
||||
|
||||
/**
|
||||
* ring_buffer_last_boot_delta - return the delta offset from last boot
|
||||
* @buffer: The buffer to return the delta from
|
||||
* @text: Return text delta
|
||||
* @data: Return data delta
|
||||
*
|
||||
* Returns: The true if the delta is non zero
|
||||
*/
|
||||
bool ring_buffer_last_boot_delta(struct trace_buffer *buffer, long *text,
|
||||
long *data)
|
||||
void *ring_buffer_meta_scratch(struct trace_buffer *buffer, unsigned int *size)
|
||||
{
|
||||
if (!buffer)
|
||||
return false;
|
||||
struct ring_buffer_meta *meta;
|
||||
void *ptr;
|
||||
|
||||
if (!buffer->last_text_delta)
|
||||
return false;
|
||||
if (!buffer || !buffer->meta)
|
||||
return NULL;
|
||||
|
||||
*text = buffer->last_text_delta;
|
||||
*data = buffer->last_data_delta;
|
||||
meta = buffer->meta;
|
||||
|
||||
return true;
|
||||
ptr = (void *)ALIGN((unsigned long)meta + sizeof(*meta), sizeof(long));
|
||||
|
||||
if (size)
|
||||
*size = (void *)meta + meta->buffers_offset - ptr;
|
||||
|
||||
return ptr;
|
||||
}
|
||||
|
||||
/**
|
||||
@ -3105,7 +3158,7 @@ static void rb_inc_iter(struct ring_buffer_iter *iter)
|
||||
}
|
||||
|
||||
/* Return the index into the sub-buffers for a given sub-buffer */
|
||||
static int rb_meta_subbuf_idx(struct ring_buffer_meta *meta, void *subbuf)
|
||||
static int rb_meta_subbuf_idx(struct ring_buffer_cpu_meta *meta, void *subbuf)
|
||||
{
|
||||
void *subbuf_array;
|
||||
|
||||
@ -3117,7 +3170,7 @@ static int rb_meta_subbuf_idx(struct ring_buffer_meta *meta, void *subbuf)
|
||||
static void rb_update_meta_head(struct ring_buffer_per_cpu *cpu_buffer,
|
||||
struct buffer_page *next_page)
|
||||
{
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
unsigned long old_head = (unsigned long)next_page->page;
|
||||
unsigned long new_head;
|
||||
|
||||
@ -3134,7 +3187,7 @@ static void rb_update_meta_head(struct ring_buffer_per_cpu *cpu_buffer,
|
||||
static void rb_update_meta_reader(struct ring_buffer_per_cpu *cpu_buffer,
|
||||
struct buffer_page *reader)
|
||||
{
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
void *old_reader = cpu_buffer->reader_page->page;
|
||||
void *new_reader = reader->page;
|
||||
int id;
|
||||
@ -3763,7 +3816,7 @@ rb_set_commit_to_write(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
rb_page_write(cpu_buffer->commit_page));
|
||||
rb_inc_page(&cpu_buffer->commit_page);
|
||||
if (cpu_buffer->ring_meta) {
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
meta->commit_buffer = (unsigned long)cpu_buffer->commit_page->page;
|
||||
}
|
||||
/* add barrier to keep gcc from optimizing too much */
|
||||
@ -6016,7 +6069,7 @@ rb_reset_cpu(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
if (cpu_buffer->mapped) {
|
||||
rb_update_meta_page(cpu_buffer);
|
||||
if (cpu_buffer->ring_meta) {
|
||||
struct ring_buffer_meta *meta = cpu_buffer->ring_meta;
|
||||
struct ring_buffer_cpu_meta *meta = cpu_buffer->ring_meta;
|
||||
meta->commit_buffer = meta->head_buffer;
|
||||
}
|
||||
}
|
||||
@ -6050,7 +6103,6 @@ static void reset_disabled_cpu_buffer(struct ring_buffer_per_cpu *cpu_buffer)
|
||||
void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
|
||||
{
|
||||
struct ring_buffer_per_cpu *cpu_buffer = buffer->buffers[cpu];
|
||||
struct ring_buffer_meta *meta;
|
||||
|
||||
if (!cpumask_test_cpu(cpu, buffer->cpumask))
|
||||
return;
|
||||
@ -6069,11 +6121,6 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
|
||||
atomic_dec(&cpu_buffer->record_disabled);
|
||||
atomic_dec(&cpu_buffer->resize_disabled);
|
||||
|
||||
/* Make sure persistent meta now uses this buffer's addresses */
|
||||
meta = rb_range_meta(buffer, 0, cpu_buffer->cpu);
|
||||
if (meta)
|
||||
rb_meta_init_text_addr(meta);
|
||||
|
||||
mutex_unlock(&buffer->mutex);
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(ring_buffer_reset_cpu);
|
||||
@ -6088,7 +6135,6 @@ EXPORT_SYMBOL_GPL(ring_buffer_reset_cpu);
|
||||
void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
|
||||
{
|
||||
struct ring_buffer_per_cpu *cpu_buffer;
|
||||
struct ring_buffer_meta *meta;
|
||||
int cpu;
|
||||
|
||||
/* prevent another thread from changing buffer sizes */
|
||||
@ -6116,11 +6162,6 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
|
||||
|
||||
reset_disabled_cpu_buffer(cpu_buffer);
|
||||
|
||||
/* Make sure persistent meta now uses this buffer's addresses */
|
||||
meta = rb_range_meta(buffer, 0, cpu_buffer->cpu);
|
||||
if (meta)
|
||||
rb_meta_init_text_addr(meta);
|
||||
|
||||
atomic_dec(&cpu_buffer->record_disabled);
|
||||
atomic_sub(RESET_BIT, &cpu_buffer->resize_disabled);
|
||||
}
|
||||
@ -7411,9 +7452,9 @@ static __init int rb_write_something(struct rb_test_data *data, bool nested)
|
||||
/* Ignore dropped events before test starts. */
|
||||
if (started) {
|
||||
if (nested)
|
||||
data->bytes_dropped += len;
|
||||
else
|
||||
data->bytes_dropped_nested += len;
|
||||
else
|
||||
data->bytes_dropped += len;
|
||||
}
|
||||
return len;
|
||||
}
|
||||
|
||||
@ -49,6 +49,7 @@
|
||||
#include <linux/fsnotify.h>
|
||||
#include <linux/irq_work.h>
|
||||
#include <linux/workqueue.h>
|
||||
#include <linux/sort.h>
|
||||
|
||||
#include <asm/setup.h> /* COMMAND_LINE_SIZE */
|
||||
|
||||
@ -4206,7 +4207,7 @@ static enum print_line_t print_trace_fmt(struct trace_iterator *iter)
|
||||
* safe to use if the array has delta offsets
|
||||
* Force printing via the fields.
|
||||
*/
|
||||
if ((tr->text_delta || tr->data_delta) &&
|
||||
if ((tr->text_delta) &&
|
||||
event->type > __TRACE_LAST_TYPE)
|
||||
return print_event_fields(iter, event);
|
||||
|
||||
@ -6001,11 +6002,130 @@ ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
|
||||
return __tracing_resize_ring_buffer(tr, size, cpu_id);
|
||||
}
|
||||
|
||||
struct trace_mod_entry {
|
||||
unsigned long mod_addr;
|
||||
char mod_name[MODULE_NAME_LEN];
|
||||
};
|
||||
|
||||
struct trace_scratch {
|
||||
unsigned long text_addr;
|
||||
unsigned long nr_entries;
|
||||
struct trace_mod_entry entries[];
|
||||
};
|
||||
|
||||
static DEFINE_MUTEX(scratch_mutex);
|
||||
|
||||
static int cmp_mod_entry(const void *key, const void *pivot)
|
||||
{
|
||||
unsigned long addr = (unsigned long)key;
|
||||
const struct trace_mod_entry *ent = pivot;
|
||||
|
||||
if (addr >= ent[0].mod_addr && addr < ent[1].mod_addr)
|
||||
return 0;
|
||||
else
|
||||
return addr - ent->mod_addr;
|
||||
}
|
||||
|
||||
/**
|
||||
* trace_adjust_address() - Adjust prev boot address to current address.
|
||||
* @tr: Persistent ring buffer's trace_array.
|
||||
* @addr: Address in @tr which is adjusted.
|
||||
*/
|
||||
unsigned long trace_adjust_address(struct trace_array *tr, unsigned long addr)
|
||||
{
|
||||
struct trace_module_delta *module_delta;
|
||||
struct trace_scratch *tscratch;
|
||||
struct trace_mod_entry *entry;
|
||||
int idx = 0, nr_entries;
|
||||
|
||||
/* If we don't have last boot delta, return the address */
|
||||
if (!(tr->flags & TRACE_ARRAY_FL_LAST_BOOT))
|
||||
return addr;
|
||||
|
||||
/* tr->module_delta must be protected by rcu. */
|
||||
guard(rcu)();
|
||||
tscratch = tr->scratch;
|
||||
/* if there is no tscrach, module_delta must be NULL. */
|
||||
module_delta = READ_ONCE(tr->module_delta);
|
||||
if (!module_delta || tscratch->entries[0].mod_addr > addr)
|
||||
return addr + tr->text_delta;
|
||||
|
||||
/* Note that entries must be sorted. */
|
||||
nr_entries = tscratch->nr_entries;
|
||||
if (nr_entries == 1 ||
|
||||
tscratch->entries[nr_entries - 1].mod_addr < addr)
|
||||
idx = nr_entries - 1;
|
||||
else {
|
||||
entry = __inline_bsearch((void *)addr,
|
||||
tscratch->entries,
|
||||
nr_entries - 1,
|
||||
sizeof(tscratch->entries[0]),
|
||||
cmp_mod_entry);
|
||||
if (entry)
|
||||
idx = entry - tscratch->entries;
|
||||
}
|
||||
|
||||
return addr + module_delta->delta[idx];
|
||||
}
|
||||
|
||||
#ifdef CONFIG_MODULES
|
||||
static int save_mod(struct module *mod, void *data)
|
||||
{
|
||||
struct trace_array *tr = data;
|
||||
struct trace_scratch *tscratch;
|
||||
struct trace_mod_entry *entry;
|
||||
unsigned int size;
|
||||
|
||||
tscratch = tr->scratch;
|
||||
if (!tscratch)
|
||||
return -1;
|
||||
size = tr->scratch_size;
|
||||
|
||||
if (struct_size(tscratch, entries, tscratch->nr_entries + 1) > size)
|
||||
return -1;
|
||||
|
||||
entry = &tscratch->entries[tscratch->nr_entries];
|
||||
|
||||
tscratch->nr_entries++;
|
||||
|
||||
entry->mod_addr = (unsigned long)mod->mem[MOD_TEXT].base;
|
||||
strscpy(entry->mod_name, mod->name);
|
||||
|
||||
return 0;
|
||||
}
|
||||
#else
|
||||
static int save_mod(struct module *mod, void *data)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
static void update_last_data(struct trace_array *tr)
|
||||
{
|
||||
if (!tr->text_delta && !tr->data_delta)
|
||||
struct trace_module_delta *module_delta;
|
||||
struct trace_scratch *tscratch;
|
||||
|
||||
if (!(tr->flags & TRACE_ARRAY_FL_BOOT))
|
||||
return;
|
||||
|
||||
if (!(tr->flags & TRACE_ARRAY_FL_LAST_BOOT))
|
||||
return;
|
||||
|
||||
/* Only if the buffer has previous boot data clear and update it. */
|
||||
tr->flags &= ~TRACE_ARRAY_FL_LAST_BOOT;
|
||||
|
||||
/* Reset the module list and reload them */
|
||||
if (tr->scratch) {
|
||||
struct trace_scratch *tscratch = tr->scratch;
|
||||
|
||||
memset(tscratch->entries, 0,
|
||||
flex_array_size(tscratch, entries, tscratch->nr_entries));
|
||||
tscratch->nr_entries = 0;
|
||||
|
||||
guard(mutex)(&scratch_mutex);
|
||||
module_for_each_mod(save_mod, tr);
|
||||
}
|
||||
|
||||
/*
|
||||
* Need to clear all CPU buffers as there cannot be events
|
||||
* from the previous boot mixed with events with this boot
|
||||
@ -6016,7 +6136,17 @@ static void update_last_data(struct trace_array *tr)
|
||||
|
||||
/* Using current data now */
|
||||
tr->text_delta = 0;
|
||||
tr->data_delta = 0;
|
||||
|
||||
if (!tr->scratch)
|
||||
return;
|
||||
|
||||
tscratch = tr->scratch;
|
||||
module_delta = READ_ONCE(tr->module_delta);
|
||||
WRITE_ONCE(tr->module_delta, NULL);
|
||||
kfree_rcu(module_delta, rcu);
|
||||
|
||||
/* Set the persistent ring buffer meta data to this address */
|
||||
tscratch->text_addr = (unsigned long)_text;
|
||||
}
|
||||
|
||||
/**
|
||||
@ -6825,19 +6955,102 @@ tracing_total_entries_read(struct file *filp, char __user *ubuf,
|
||||
return simple_read_from_buffer(ubuf, cnt, ppos, buf, r);
|
||||
}
|
||||
|
||||
static ssize_t
|
||||
tracing_last_boot_read(struct file *filp, char __user *ubuf, size_t cnt, loff_t *ppos)
|
||||
#define LAST_BOOT_HEADER ((void *)1)
|
||||
|
||||
static void *l_next(struct seq_file *m, void *v, loff_t *pos)
|
||||
{
|
||||
struct trace_array *tr = filp->private_data;
|
||||
struct seq_buf seq;
|
||||
char buf[64];
|
||||
struct trace_array *tr = m->private;
|
||||
struct trace_scratch *tscratch = tr->scratch;
|
||||
unsigned int index = *pos;
|
||||
|
||||
seq_buf_init(&seq, buf, 64);
|
||||
(*pos)++;
|
||||
|
||||
seq_buf_printf(&seq, "text delta:\t%ld\n", tr->text_delta);
|
||||
seq_buf_printf(&seq, "data delta:\t%ld\n", tr->data_delta);
|
||||
if (*pos == 1)
|
||||
return LAST_BOOT_HEADER;
|
||||
|
||||
return simple_read_from_buffer(ubuf, cnt, ppos, buf, seq_buf_used(&seq));
|
||||
/* Only show offsets of the last boot data */
|
||||
if (!tscratch || !(tr->flags & TRACE_ARRAY_FL_LAST_BOOT))
|
||||
return NULL;
|
||||
|
||||
/* *pos 0 is for the header, 1 is for the first module */
|
||||
index--;
|
||||
|
||||
if (index >= tscratch->nr_entries)
|
||||
return NULL;
|
||||
|
||||
return &tscratch->entries[index];
|
||||
}
|
||||
|
||||
static void *l_start(struct seq_file *m, loff_t *pos)
|
||||
{
|
||||
mutex_lock(&scratch_mutex);
|
||||
|
||||
return l_next(m, NULL, pos);
|
||||
}
|
||||
|
||||
static void l_stop(struct seq_file *m, void *p)
|
||||
{
|
||||
mutex_unlock(&scratch_mutex);
|
||||
}
|
||||
|
||||
static void show_last_boot_header(struct seq_file *m, struct trace_array *tr)
|
||||
{
|
||||
struct trace_scratch *tscratch = tr->scratch;
|
||||
|
||||
/*
|
||||
* Do not leak KASLR address. This only shows the KASLR address of
|
||||
* the last boot. When the ring buffer is started, the LAST_BOOT
|
||||
* flag gets cleared, and this should only report "current".
|
||||
* Otherwise it shows the KASLR address from the previous boot which
|
||||
* should not be the same as the current boot.
|
||||
*/
|
||||
if (tscratch && (tr->flags & TRACE_ARRAY_FL_LAST_BOOT))
|
||||
seq_printf(m, "%lx\t[kernel]\n", tscratch->text_addr);
|
||||
else
|
||||
seq_puts(m, "# Current\n");
|
||||
}
|
||||
|
||||
static int l_show(struct seq_file *m, void *v)
|
||||
{
|
||||
struct trace_array *tr = m->private;
|
||||
struct trace_mod_entry *entry = v;
|
||||
|
||||
if (v == LAST_BOOT_HEADER) {
|
||||
show_last_boot_header(m, tr);
|
||||
return 0;
|
||||
}
|
||||
|
||||
seq_printf(m, "%lx\t%s\n", entry->mod_addr, entry->mod_name);
|
||||
return 0;
|
||||
}
|
||||
|
||||
static const struct seq_operations last_boot_seq_ops = {
|
||||
.start = l_start,
|
||||
.next = l_next,
|
||||
.stop = l_stop,
|
||||
.show = l_show,
|
||||
};
|
||||
|
||||
static int tracing_last_boot_open(struct inode *inode, struct file *file)
|
||||
{
|
||||
struct trace_array *tr = inode->i_private;
|
||||
struct seq_file *m;
|
||||
int ret;
|
||||
|
||||
ret = tracing_check_open_get_tr(tr);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = seq_open(file, &last_boot_seq_ops);
|
||||
if (ret) {
|
||||
trace_array_put(tr);
|
||||
return ret;
|
||||
}
|
||||
|
||||
m = file->private_data;
|
||||
m->private = tr;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int tracing_buffer_meta_open(struct inode *inode, struct file *filp)
|
||||
@ -7466,10 +7679,10 @@ static const struct file_operations trace_time_stamp_mode_fops = {
|
||||
};
|
||||
|
||||
static const struct file_operations last_boot_fops = {
|
||||
.open = tracing_open_generic_tr,
|
||||
.read = tracing_last_boot_read,
|
||||
.llseek = generic_file_llseek,
|
||||
.release = tracing_release_generic_tr,
|
||||
.open = tracing_last_boot_open,
|
||||
.read = seq_read,
|
||||
.llseek = seq_lseek,
|
||||
.release = tracing_seq_release,
|
||||
};
|
||||
|
||||
#ifdef CONFIG_TRACER_SNAPSHOT
|
||||
@ -9209,22 +9422,125 @@ static struct dentry *trace_instance_dir;
|
||||
static void
|
||||
init_tracer_tracefs(struct trace_array *tr, struct dentry *d_tracer);
|
||||
|
||||
#ifdef CONFIG_MODULES
|
||||
static int make_mod_delta(struct module *mod, void *data)
|
||||
{
|
||||
struct trace_module_delta *module_delta;
|
||||
struct trace_scratch *tscratch;
|
||||
struct trace_mod_entry *entry;
|
||||
struct trace_array *tr = data;
|
||||
int i;
|
||||
|
||||
tscratch = tr->scratch;
|
||||
module_delta = READ_ONCE(tr->module_delta);
|
||||
for (i = 0; i < tscratch->nr_entries; i++) {
|
||||
entry = &tscratch->entries[i];
|
||||
if (strcmp(mod->name, entry->mod_name))
|
||||
continue;
|
||||
if (mod->state == MODULE_STATE_GOING)
|
||||
module_delta->delta[i] = 0;
|
||||
else
|
||||
module_delta->delta[i] = (unsigned long)mod->mem[MOD_TEXT].base
|
||||
- entry->mod_addr;
|
||||
break;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
#else
|
||||
static int make_mod_delta(struct module *mod, void *data)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
static int mod_addr_comp(const void *a, const void *b, const void *data)
|
||||
{
|
||||
const struct trace_mod_entry *e1 = a;
|
||||
const struct trace_mod_entry *e2 = b;
|
||||
|
||||
return e1->mod_addr > e2->mod_addr ? 1 : -1;
|
||||
}
|
||||
|
||||
static void setup_trace_scratch(struct trace_array *tr,
|
||||
struct trace_scratch *tscratch, unsigned int size)
|
||||
{
|
||||
struct trace_module_delta *module_delta;
|
||||
struct trace_mod_entry *entry;
|
||||
int i, nr_entries;
|
||||
|
||||
if (!tscratch)
|
||||
return;
|
||||
|
||||
tr->scratch = tscratch;
|
||||
tr->scratch_size = size;
|
||||
|
||||
if (tscratch->text_addr)
|
||||
tr->text_delta = (unsigned long)_text - tscratch->text_addr;
|
||||
|
||||
if (struct_size(tscratch, entries, tscratch->nr_entries) > size)
|
||||
goto reset;
|
||||
|
||||
/* Check if each module name is a valid string */
|
||||
for (i = 0; i < tscratch->nr_entries; i++) {
|
||||
int n;
|
||||
|
||||
entry = &tscratch->entries[i];
|
||||
|
||||
for (n = 0; n < MODULE_NAME_LEN; n++) {
|
||||
if (entry->mod_name[n] == '\0')
|
||||
break;
|
||||
if (!isprint(entry->mod_name[n]))
|
||||
goto reset;
|
||||
}
|
||||
if (n == MODULE_NAME_LEN)
|
||||
goto reset;
|
||||
}
|
||||
|
||||
/* Sort the entries so that we can find appropriate module from address. */
|
||||
nr_entries = tscratch->nr_entries;
|
||||
sort_r(tscratch->entries, nr_entries, sizeof(struct trace_mod_entry),
|
||||
mod_addr_comp, NULL, NULL);
|
||||
|
||||
if (IS_ENABLED(CONFIG_MODULES)) {
|
||||
module_delta = kzalloc(struct_size(module_delta, delta, nr_entries), GFP_KERNEL);
|
||||
if (!module_delta) {
|
||||
pr_info("module_delta allocation failed. Not able to decode module address.");
|
||||
goto reset;
|
||||
}
|
||||
init_rcu_head(&module_delta->rcu);
|
||||
} else
|
||||
module_delta = NULL;
|
||||
WRITE_ONCE(tr->module_delta, module_delta);
|
||||
|
||||
/* Scan modules to make text delta for modules. */
|
||||
module_for_each_mod(make_mod_delta, tr);
|
||||
return;
|
||||
reset:
|
||||
/* Invalid trace modules */
|
||||
memset(tscratch, 0, size);
|
||||
}
|
||||
|
||||
static int
|
||||
allocate_trace_buffer(struct trace_array *tr, struct array_buffer *buf, int size)
|
||||
{
|
||||
enum ring_buffer_flags rb_flags;
|
||||
struct trace_scratch *tscratch;
|
||||
unsigned int scratch_size = 0;
|
||||
|
||||
rb_flags = tr->trace_flags & TRACE_ITER_OVERWRITE ? RB_FL_OVERWRITE : 0;
|
||||
|
||||
buf->tr = tr;
|
||||
|
||||
if (tr->range_addr_start && tr->range_addr_size) {
|
||||
/* Add scratch buffer to handle 128 modules */
|
||||
buf->buffer = ring_buffer_alloc_range(size, rb_flags, 0,
|
||||
tr->range_addr_start,
|
||||
tr->range_addr_size);
|
||||
tr->range_addr_size,
|
||||
struct_size(tscratch, entries, 128));
|
||||
|
||||
tscratch = ring_buffer_meta_scratch(buf->buffer, &scratch_size);
|
||||
setup_trace_scratch(tr, tscratch, scratch_size);
|
||||
|
||||
ring_buffer_last_boot_delta(buf->buffer,
|
||||
&tr->text_delta, &tr->data_delta);
|
||||
/*
|
||||
* This is basically the same as a mapped buffer,
|
||||
* with the same restrictions.
|
||||
@ -9297,6 +9613,9 @@ static void free_trace_buffers(struct trace_array *tr)
|
||||
#ifdef CONFIG_TRACER_MAX_TRACE
|
||||
free_trace_buffer(&tr->max_buffer);
|
||||
#endif
|
||||
|
||||
if (tr->range_addr_start)
|
||||
vunmap((void *)tr->range_addr_start);
|
||||
}
|
||||
|
||||
static void init_trace_flags_index(struct trace_array *tr)
|
||||
@ -9458,6 +9777,7 @@ trace_array_create_systems(const char *name, const char *systems,
|
||||
free_cpumask_var(tr->pipe_cpumask);
|
||||
free_cpumask_var(tr->tracing_cpumask);
|
||||
kfree_const(tr->system_names);
|
||||
kfree(tr->range_name);
|
||||
kfree(tr->name);
|
||||
kfree(tr);
|
||||
|
||||
@ -9584,6 +9904,11 @@ static int __remove_instance(struct trace_array *tr)
|
||||
free_trace_buffers(tr);
|
||||
clear_tracing_err_log(tr);
|
||||
|
||||
if (tr->range_name) {
|
||||
reserve_mem_release_by_name(tr->range_name);
|
||||
kfree(tr->range_name);
|
||||
}
|
||||
|
||||
for (i = 0; i < tr->nr_topts; i++) {
|
||||
kfree(tr->topts[i].topts);
|
||||
}
|
||||
@ -9905,6 +10230,24 @@ static void trace_module_remove_evals(struct module *mod)
|
||||
static inline void trace_module_remove_evals(struct module *mod) { }
|
||||
#endif /* CONFIG_TRACE_EVAL_MAP_FILE */
|
||||
|
||||
static void trace_module_record(struct module *mod, bool add)
|
||||
{
|
||||
struct trace_array *tr;
|
||||
unsigned long flags;
|
||||
|
||||
list_for_each_entry(tr, &ftrace_trace_arrays, list) {
|
||||
flags = tr->flags & (TRACE_ARRAY_FL_BOOT | TRACE_ARRAY_FL_LAST_BOOT);
|
||||
/* Update any persistent trace array that has already been started */
|
||||
if (flags == TRACE_ARRAY_FL_BOOT && add) {
|
||||
guard(mutex)(&scratch_mutex);
|
||||
save_mod(mod, tr);
|
||||
} else if (flags & TRACE_ARRAY_FL_LAST_BOOT) {
|
||||
/* Update delta if the module loaded in previous boot */
|
||||
make_mod_delta(mod, tr);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static int trace_module_notify(struct notifier_block *self,
|
||||
unsigned long val, void *data)
|
||||
{
|
||||
@ -9913,9 +10256,11 @@ static int trace_module_notify(struct notifier_block *self,
|
||||
switch (val) {
|
||||
case MODULE_STATE_COMING:
|
||||
trace_module_add_evals(mod);
|
||||
trace_module_record(mod, true);
|
||||
break;
|
||||
case MODULE_STATE_GOING:
|
||||
trace_module_remove_evals(mod);
|
||||
trace_module_record(mod, false);
|
||||
break;
|
||||
}
|
||||
|
||||
@ -10381,6 +10726,7 @@ __init static void enable_instances(void)
|
||||
bool traceoff = false;
|
||||
char *flag_delim;
|
||||
char *addr_delim;
|
||||
char *rname __free(kfree) = NULL;
|
||||
|
||||
tok = strsep(&curr_str, ",");
|
||||
|
||||
@ -10437,6 +10783,7 @@ __init static void enable_instances(void)
|
||||
pr_warn("Failed to map boot instance %s to %s\n", name, tok);
|
||||
continue;
|
||||
}
|
||||
rname = kstrdup(tok, GFP_KERNEL);
|
||||
}
|
||||
|
||||
if (start) {
|
||||
@ -10472,8 +10819,8 @@ __init static void enable_instances(void)
|
||||
* to it.
|
||||
*/
|
||||
if (start) {
|
||||
tr->flags |= TRACE_ARRAY_FL_BOOT;
|
||||
tr->ref++;
|
||||
tr->flags |= TRACE_ARRAY_FL_BOOT | TRACE_ARRAY_FL_LAST_BOOT;
|
||||
tr->range_name = no_free_ptr(rname);
|
||||
}
|
||||
|
||||
while ((tok = strsep(&curr_str, ","))) {
|
||||
|
||||
@ -313,6 +313,11 @@ struct trace_func_repeats {
|
||||
u64 ts_last_call;
|
||||
};
|
||||
|
||||
struct trace_module_delta {
|
||||
struct rcu_head rcu;
|
||||
long delta[];
|
||||
};
|
||||
|
||||
/*
|
||||
* The trace array - an array of per-CPU trace arrays. This is the
|
||||
* highest level data structure that individual tracers deal with.
|
||||
@ -349,8 +354,13 @@ struct trace_array {
|
||||
unsigned int mapped;
|
||||
unsigned long range_addr_start;
|
||||
unsigned long range_addr_size;
|
||||
char *range_name;
|
||||
long text_delta;
|
||||
long data_delta;
|
||||
struct trace_module_delta *module_delta;
|
||||
void *scratch; /* pointer in persistent memory */
|
||||
int scratch_size;
|
||||
|
||||
int buffer_disabled;
|
||||
|
||||
struct trace_pid_list __rcu *filtered_pids;
|
||||
struct trace_pid_list __rcu *filtered_no_pids;
|
||||
@ -368,7 +378,6 @@ struct trace_array {
|
||||
* CONFIG_TRACER_MAX_TRACE.
|
||||
*/
|
||||
arch_spinlock_t max_lock;
|
||||
int buffer_disabled;
|
||||
#ifdef CONFIG_FTRACE_SYSCALLS
|
||||
int sys_refcount_enter;
|
||||
int sys_refcount_exit;
|
||||
@ -434,9 +443,10 @@ struct trace_array {
|
||||
};
|
||||
|
||||
enum {
|
||||
TRACE_ARRAY_FL_GLOBAL = BIT(0),
|
||||
TRACE_ARRAY_FL_BOOT = BIT(1),
|
||||
TRACE_ARRAY_FL_MOD_INIT = BIT(2),
|
||||
TRACE_ARRAY_FL_GLOBAL = BIT(0),
|
||||
TRACE_ARRAY_FL_BOOT = BIT(1),
|
||||
TRACE_ARRAY_FL_LAST_BOOT = BIT(2),
|
||||
TRACE_ARRAY_FL_MOD_INIT = BIT(3),
|
||||
};
|
||||
|
||||
#ifdef CONFIG_MODULES
|
||||
@ -463,6 +473,8 @@ extern int tracing_set_clock(struct trace_array *tr, const char *clockstr);
|
||||
|
||||
extern bool trace_clock_in_ns(struct trace_array *tr);
|
||||
|
||||
extern unsigned long trace_adjust_address(struct trace_array *tr, unsigned long addr);
|
||||
|
||||
/*
|
||||
* The global tracer (top) should be the first trace array added,
|
||||
* but we check the flag anyway.
|
||||
@ -785,6 +797,8 @@ extern void trace_find_cmdline(int pid, char comm[]);
|
||||
extern int trace_find_tgid(int pid);
|
||||
extern void trace_event_follow_fork(struct trace_array *tr, bool enable);
|
||||
|
||||
extern int trace_events_enabled(struct trace_array *tr, const char *system);
|
||||
|
||||
#ifdef CONFIG_DYNAMIC_FTRACE
|
||||
extern unsigned long ftrace_update_tot_cnt;
|
||||
extern unsigned long ftrace_number_of_pages;
|
||||
|
||||
@ -1820,28 +1820,28 @@ event_enable_write(struct file *filp, const char __user *ubuf, size_t cnt,
|
||||
return cnt;
|
||||
}
|
||||
|
||||
static ssize_t
|
||||
system_enable_read(struct file *filp, char __user *ubuf, size_t cnt,
|
||||
loff_t *ppos)
|
||||
/*
|
||||
* Returns:
|
||||
* 0 : no events exist?
|
||||
* 1 : all events are disabled
|
||||
* 2 : all events are enabled
|
||||
* 3 : some events are enabled and some are enabled
|
||||
*/
|
||||
int trace_events_enabled(struct trace_array *tr, const char *system)
|
||||
{
|
||||
const char set_to_char[4] = { '?', '0', '1', 'X' };
|
||||
struct trace_subsystem_dir *dir = filp->private_data;
|
||||
struct event_subsystem *system = dir->subsystem;
|
||||
struct trace_event_call *call;
|
||||
struct trace_event_file *file;
|
||||
struct trace_array *tr = dir->tr;
|
||||
char buf[2];
|
||||
int set = 0;
|
||||
int ret;
|
||||
|
||||
mutex_lock(&event_mutex);
|
||||
guard(mutex)(&event_mutex);
|
||||
|
||||
list_for_each_entry(file, &tr->events, list) {
|
||||
call = file->event_call;
|
||||
if ((call->flags & TRACE_EVENT_FL_IGNORE_ENABLE) ||
|
||||
!trace_event_name(call) || !call->class || !call->class->reg)
|
||||
continue;
|
||||
|
||||
if (system && strcmp(call->class->system, system->name) != 0)
|
||||
if (system && strcmp(call->class->system, system) != 0)
|
||||
continue;
|
||||
|
||||
/*
|
||||
@ -1857,7 +1857,23 @@ system_enable_read(struct file *filp, char __user *ubuf, size_t cnt,
|
||||
if (set == 3)
|
||||
break;
|
||||
}
|
||||
mutex_unlock(&event_mutex);
|
||||
|
||||
return set;
|
||||
}
|
||||
|
||||
static ssize_t
|
||||
system_enable_read(struct file *filp, char __user *ubuf, size_t cnt,
|
||||
loff_t *ppos)
|
||||
{
|
||||
const char set_to_char[4] = { '?', '0', '1', 'X' };
|
||||
struct trace_subsystem_dir *dir = filp->private_data;
|
||||
struct event_subsystem *system = dir->subsystem;
|
||||
struct trace_array *tr = dir->tr;
|
||||
char buf[2];
|
||||
int set;
|
||||
int ret;
|
||||
|
||||
set = trace_events_enabled(tr, system ? system->name : NULL);
|
||||
|
||||
buf[0] = set_to_char[set];
|
||||
buf[1] = '\n';
|
||||
|
||||
@ -5,6 +5,7 @@
|
||||
* Copyright (C) 2008 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
|
||||
*
|
||||
*/
|
||||
#include "trace.h"
|
||||
#include <linux/module.h>
|
||||
#include <linux/mutex.h>
|
||||
#include <linux/ftrace.h>
|
||||
@ -1340,7 +1341,6 @@ static enum print_line_t trace_stack_print(struct trace_iterator *iter,
|
||||
struct trace_seq *s = &iter->seq;
|
||||
unsigned long *p;
|
||||
unsigned long *end;
|
||||
long delta = iter->tr->text_delta;
|
||||
|
||||
trace_assign_type(field, iter->ent);
|
||||
end = (unsigned long *)((long)iter->ent + iter->ent_size);
|
||||
@ -1357,7 +1357,7 @@ static enum print_line_t trace_stack_print(struct trace_iterator *iter,
|
||||
trace_seq_puts(s, "[FTRACE TRAMPOLINE]\n");
|
||||
continue;
|
||||
}
|
||||
seq_print_ip_sym(s, (*p) + delta, flags);
|
||||
seq_print_ip_sym(s, trace_adjust_address(iter->tr, *p), flags);
|
||||
trace_seq_putc(s, '\n');
|
||||
}
|
||||
|
||||
|
||||
@ -16,6 +16,7 @@
|
||||
#include <linux/kmemleak.h>
|
||||
#include <linux/seq_file.h>
|
||||
#include <linux/memblock.h>
|
||||
#include <linux/mutex.h>
|
||||
|
||||
#include <asm/sections.h>
|
||||
#include <linux/io.h>
|
||||
@ -2283,6 +2284,7 @@ struct reserve_mem_table {
|
||||
};
|
||||
static struct reserve_mem_table reserved_mem_table[RESERVE_MEM_MAX_ENTRIES];
|
||||
static int reserved_mem_count;
|
||||
static DEFINE_MUTEX(reserve_mem_lock);
|
||||
|
||||
/* Add wildcard region with a lookup name */
|
||||
static void __init reserved_mem_add(phys_addr_t start, phys_addr_t size,
|
||||
@ -2296,6 +2298,21 @@ static void __init reserved_mem_add(phys_addr_t start, phys_addr_t size,
|
||||
strscpy(map->name, name);
|
||||
}
|
||||
|
||||
static struct reserve_mem_table *reserve_mem_find_by_name_nolock(const char *name)
|
||||
{
|
||||
struct reserve_mem_table *map;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < reserved_mem_count; i++) {
|
||||
map = &reserved_mem_table[i];
|
||||
if (!map->size)
|
||||
continue;
|
||||
if (strcmp(name, map->name) == 0)
|
||||
return map;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* reserve_mem_find_by_name - Find reserved memory region with a given name
|
||||
* @name: The name that is attached to a reserved memory region
|
||||
@ -2309,22 +2326,47 @@ static void __init reserved_mem_add(phys_addr_t start, phys_addr_t size,
|
||||
int reserve_mem_find_by_name(const char *name, phys_addr_t *start, phys_addr_t *size)
|
||||
{
|
||||
struct reserve_mem_table *map;
|
||||
int i;
|
||||
|
||||
for (i = 0; i < reserved_mem_count; i++) {
|
||||
map = &reserved_mem_table[i];
|
||||
if (!map->size)
|
||||
continue;
|
||||
if (strcmp(name, map->name) == 0) {
|
||||
*start = map->start;
|
||||
*size = map->size;
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
guard(mutex)(&reserve_mem_lock);
|
||||
map = reserve_mem_find_by_name_nolock(name);
|
||||
if (!map)
|
||||
return 0;
|
||||
|
||||
*start = map->start;
|
||||
*size = map->size;
|
||||
return 1;
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(reserve_mem_find_by_name);
|
||||
|
||||
/**
|
||||
* reserve_mem_release_by_name - Release reserved memory region with a given name
|
||||
* @name: The name that is attatched to a reserved memory region
|
||||
*
|
||||
* Forcibly release the pages in the reserved memory region so that those memory
|
||||
* can be used as free memory. After released the reserved region size becomes 0.
|
||||
*
|
||||
* Returns: 1 if released or 0 if not found.
|
||||
*/
|
||||
int reserve_mem_release_by_name(const char *name)
|
||||
{
|
||||
char buf[RESERVE_MEM_NAME_SIZE + 12];
|
||||
struct reserve_mem_table *map;
|
||||
void *start, *end;
|
||||
|
||||
guard(mutex)(&reserve_mem_lock);
|
||||
map = reserve_mem_find_by_name_nolock(name);
|
||||
if (!map)
|
||||
return 0;
|
||||
|
||||
start = phys_to_virt(map->start);
|
||||
end = start + map->size - 1;
|
||||
snprintf(buf, sizeof(buf), "reserve_mem:%s", name);
|
||||
free_reserved_area(start, end, 0, buf);
|
||||
map->size = 0;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
/*
|
||||
* Parse reserve_mem=nn:align:name
|
||||
*/
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user