BPF Updates 12

This is issue 12 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

The highlights since the last time

• Generic metadata transfer from XDP into skb via new helper function bpf_xdp_adjust_meta.
• bpf_perf_event_read_value helper function series got merged.
• Multiple programs can now be attached to a cgroup.
• A new map type cpumap for XDP got merged.

The addition of metadata transfer from XDP allows XDP programs to apply early filters to extract metadata from packets and make it available to BPF programs later in the forwarding chain such as programs attached to clsact qdiscs.

The new values available via the perf helper make it easier to normalize measurements. cpumap let's you redirect XDP frames to remote CPU's where these can handle the packet representation and fire up the networking stack.

More interesting topics

• More fields added to bpf_prog_info.
• Netronome bpftool upstreamed.
• LPM performance improvements.
• 32-bit eBPF encoding support.

The BPF tooling is evolving and introspection is getting better.

Presentations

Talks

All Systems Go! 2017 - A gentle introduction to [e]BPF

Beginner friendly overview of BPF.

All Systems Go! 2017 - High-performance Linux monitoring with eBPF

Intro to BPF and short part on how it's being leveraged in weaveworks.

All Systems Go! 2017 - Using BPF in Kubernetes

Nice talk focusing on some of the use cases in the cloud.

Kernel Recipes 2017 - EBPF and XDP - Eric Leblond

Overview of how Suricata uses BPF Nice to see the version number for the various features / work mentioned. The slides also has some code.

Kernel Recipes 2017 - Performance Analysis with BPF - Brendan Gregg

Introduction to BPF focusing on BCC and tracing.

Slides

eBPF cgroup filters for data usage accounting on Android

Interesting work on how to use BPF for wifi data usage.

Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF

High level introduction to Cilium with some new ongoing work on Cilium. If you are using / interested in Kafka, worth checking out for the API filtering.

eBPF Implementation for FreeBSD

Interesting slides on eBPF port to FreeBSD.

In case you missed it

Heap Allocation Flamegraphs

Post on using one of the grave tools for tracing JVM processes.

Linux の BPF : (5) eBPF による Linux Kernel Tracing

Going through a tracing example using kernel 4.12.

An intro to using eBPF to filter packets in the Linux kernel

Beginner friendly post with a accompanying example for tracing.

IP Accounting and Access Lists with systemd

Simple cgroup/BPF packet counting and blacklist via systemd.

XDP on Power

XDP on the Power architecture.

Cilium v0.10 & v0.11 Released: Double the Fun - Two Updates in One!

Great post on the recent progress made in the Cilium project.

Projects

A couple of new GitHub projects. Please star if you like the projects ;)

Cilium client API example

Simple example illustrating use of the Cilium API.

ebpf_asm

An assembler for eBPF programs written in an Intel-like assembly syntax.

go-ebpf

eBPF programs without a libbcc dependency

generic-ebpf

Generic eBPF VM. Currently support FreeBSD kernel, FreeBSD userspace, Linux kernel, Linux userspace and MacOSX userspace.

Random cool note

Schedule is out! Dont miss this amazing event. Come to the land of Kimchi and Palaces. Hangout with awesome geeks. https://www.netdevconf.org/2.2/schedule.html

Patches

• Daniel Borkmann, [PATCH net 0/3] Fix for BPF devmap percpu allocation splat
  • [PATCH net 1/3] mm, percpu: add support for __GFP_NOWARN flag
  • [PATCH net 2/3] bpf: fix splat for illegal devmap percpu allocation
  • [PATCH net 3/3] bpf: do not test for PCPU_MIN_UNIT_SIZE before percpu allocations
• Richard Weinber, [PATCH 1/3] bpf: Don't check for current being NULL
  • [PATCH 2/3] bpf: Remove dead variable
  • [PATCH 3/3] bpf: Make sure that ->comm does not change under us.
• Tariq Toukan, [PATCH net-next 0/3] mlx4_en XDP TX improvements
  • [PATCH net-next 1/3] net/mlx4_en: Replace netdev parameter with priv in XDP xmit function
  • [PATCH net-next 2/3] net/mlx4_en: Obsolete call to generic write_desc in XDP xmit flow
  • [PATCH net-next 3/3] net/mlx4_en: XDP_TX, assign constant values of TX descs on ring creaion
• Chenbo Feng, [PATCH net-next v6 0/5] bpf: security: New file mode and LSM hooks for eBPF object permission control
  • [PATCH net-next v6 1/5] bpf: Add file mode configuration into bpf maps
  • [PATCH net-next v6 2/5] bpf: Add tests for eBPF file mode
  • [PATCH net-next v6 3/5] security: bpf: Add LSM hooks for bpf object related syscall
  • [PATCH net-next v6 4/5] selinux: bpf: Add selinux check for eBPF syscall operations
  • [PATCH net-next v6 5/5] selinux: bpf: Add addtional check for bpf object file receive
• Jakub Kicinski, [PATCH net-next 0/4] bpf: move context info out of the verifier
  • [PATCH net-next 1/4] bpf: split verifier and program ops
  • [PATCH net-next 2/4] bpf: remove the verifier ops from program structure
  • [PATCH net-next 3/4] bpf: move knowledge about post-translation offsets out of verifier
  • [PATCH net-next 4/4] bpf: allow access to skb->len from offloads
• Jakub Kicinski, [PATCH net v2] bpf: disallow arithmetic operations on context pointer
• Jakub Kicinski, [PATCH net-next] tools: bpftool: use more common tag format
• Jakub Kicinski, [PATCH net-next 00/12] nfp: bpf: support direct packet access
  • [PATCH net-next 01/12] bpf: verifier: set reg_type on context accesses in second pass
  • [PATCH net-next 02/12] nfp: bpf: reorder arguments to emit_ld_field_any()
  • [PATCH net-next 03/12] nfp: bpf: add missing return in jne_imm optimization
  • [PATCH net-next 04/12] nfp: bpf: fix compare instructions
  • [PATCH net-next 05/12] nfp: bpf: add mov helper
  • [PATCH net-next 06/12] nfp: bpf: implement byte swap instruction
  • [PATCH net-next 07/12] nfp: bpf: support BPF offload only on little endian
  • [PATCH net-next 08/12] nfp: bpf: fix context accesses
  • [PATCH net-next 09/12] nfp: bpf: separate I/O from checks for legacy data load
  • [PATCH net-next 10/12] nfp: bpf: add support for direct packet access - read
  • [PATCH net-next 11/12] nfp: bpf: direct packet access - write
  • [PATCH net-next 12/12] nfp: bpf: support direct packet access in TC
• Jesper Dangaard Brouer, [net-next V8 PATCH 0/5] New bpf cpumap type for XDP_REDIRECT
  • [net-next V8 PATCH 1/5] bpf: introduce new bpf cpu map type BPF_MAP_TYPE_CPUMAP
  • [net-next V8 PATCH 2/5] bpf: XDP_REDIRECT enable use of cpumap
  • [net-next V8 PATCH 3/5] bpf: cpumap xdp_buff to skb conversion and allocation
  • [net-next V8 PATCH 4/5] bpf: cpumap add tracepoints
  • [net-next V8 PATCH 5/5] samples/bpf: add cpumap sample program xdp_redirect_cpu
• Jakub Kicinski, [PATCH net-next v2 0/7] bpf: get rid of global verifier state and reuse instruction printer
  • [PATCH net-next v2 1/7] selftests/bpf: add a test for verifier logs
  • [PATCH net-next v2 2/7] bpf: encapsulate verifier log state into a structure
  • [PATCH net-next v2 3/7] bpf: move global verifier log into verifier environment
  • [PATCH net-next v2 4/7] bpf: move instruction printing into a separate file
  • [PATCH net-next v2 5/7] tools: bpftool: use the kernel's instruction printer
  • [PATCH net-next v2 6/7] bpf: don't rely on the verifier lock for metadata_dst allocation
  • [PATCH net-next v2 7/7] bpf: write back the verifier log buffer as it gets filled
• Jiong Wang, [PATCH, bpf-llvm] Fix bug on silently truncating 64-bit immediate
• Richard Weinberger, [PATCH] bpf: devmap: Check attr->max_entries more carefully
• Colin King, [PATCH][bpf-next] bpf: remove redundant variable old_flags
• Steven Rostedt, [PATCH] tracing: bpf: Hide bpf trace events when they are not used
• Jakub Kicinski, [PATCH net-next 00/15] nfp: bpf ABIv2 and multi port
  • [PATCH net-next 01/15] nfp: output control messages to trace_devlink_hwmsg()
  • [PATCH net-next 02/15] nfp: bpf: lift the single-port limitation
  • [PATCH net-next 03/15] nfp: bpf: use the power of sparse to check we encode registers right
  • [PATCH net-next 04/15] nfp: bpf: move software reg helpers and cmd table out of translator
  • [PATCH net-next 05/15] nfp: bpf: encode all 64bit shifts
  • [PATCH net-next 06/15] nfp: bpf: remove register rename
  • [PATCH net-next 07/15] nfp: bpf: remove packet marking support
  • [PATCH net-next 08/15] nfp: add more white space to the instruction defines
  • [PATCH net-next 09/15] nfp: bpf: encode LMEM accesses
  • [PATCH net-next 10/15] nfp: bpf: encode extended LM pointer operands
  • [PATCH net-next 11/15] nfp: bpf: move to datapath ABI version 2
  • [PATCH net-next 12/15] nfp: bpf: calculate code store ECC
  • [PATCH net-next 13/15] nfp: bpf: pad code with valid nops
  • [PATCH net-next 14/15] nfp: bpf: byte swap the instructions
  • [PATCH net-next 15/15] nfp: bpf: pass dst register to ld_field instruction
• Yonghong Song, [llvm] r315889 - bpf: fix bug on silently truncating 64-bit immediate
• Shmulik Ladkani, [PATCH v2] netfilter: xt_bpf: Fix XT_BPF_MODE_FD_PINNED mode of 'xt_bpf_info_v1'
• Christina Jacob, [PATCH v2] XDP Program for Ip forward
  • [PATCH 1/1] xdp: Sample xdp program implementing ip forward
• Martin KaFai Lau, [PATCH net-next 0/3] bpf: Misc improvements and a new usage on bpf obj name
  • [PATCH net-next 1/3] bpf: Change bpf_obj_name_cpy() to better ensure map's name is init by 0
  • [PATCH net-next 2/3] bpf: Use char in prog and map name
  • [PATCH net-next 3/3] bpf: Append prog->aux->name in bpf_get_prog_name()
• Yonghong Song, [PATCH net-next v7 0/5] bpf: add two helpers to read perf event enabled/running time
  • [PATCH net-next v7 1/5] bpf: perf event change needed for subsequent bpf helpers
  • [PATCH net-next v7 2/5] bpf: add helper bpf_perf_event_read_value for perf event array map
  • [PATCH net-next v7 3/5] bpf: add a test case for helper bpf_perf_event_read_value
  • [PATCH net-next v7 4/5] bpf: add helper bpf_perf_prog_read_value
  • [PATCH net-next v7 5/5] bpf: add a test case for helper bpf_perf_prog_read_value
• Craig Gallek, [PATCH net-next v3 0/2] libbpf: support more map options
  • [PATCH net-next v3 1/2] libbpf: parse maps sections of varying size
  • [PATCH net-next v3 2/2] libbpf: use map_flags when creating maps
• Jesper Dangaard Brouer, [net-next PATCH 0/3] Improve xdp_monitor samples/bpf
  • [net-next PATCH 1/3] samples/bpf: xdp_monitor first 8 bytes are not accessible by bpf
  • [net-next PATCH 2/3] samples/bpf: xdp_monitor also record xdp_exception tracepoint
  • [net-next PATCH 3/3] samples/bpf: xdp_monitor increase memory rlimit
• Alexei Starovoitov, [PATCH net] bpf: fix liveness marking
• Alexei Starovoitov, [PATCH v2 net-next 0/8] bpf: muli prog support for cgroup-bpf
  • [PATCH v2 net-next 1/8] bpf: multi program support for cgroup+bpf
  • [PATCH v2 net-next 2/8] bpf: introduce BPF_PROG_QUERY command
  • [PATCH v2 net-next 3/8] bpf: enforce return code for cgroup-bpf programs
  • [PATCH v2 net-next 4/8] libbpf: introduce bpf_prog_detach2()
  • [PATCH v2 net-next 5/8] samples/bpf: add multi-prog cgroup test case
  • [PATCH v2 net-next 6/8] libbpf: sync bpf.h
  • [PATCH v2 net-next 7/8] libbpf: add support for BPF_PROG_QUERY
  • [PATCH v2 net-next 8/8] samples/bpf: use bpf_prog_query() interface
• Yonghong Song, [llvm] r314911 - bpf: fix an insn encoding issue for neg insn
• Alexei Starovoitov, [PATCH net] bpf: fix bpf_tail_call() x64 JIT
• Jakub Kicinski, [PATCH net-next v4 0/3] tools: add bpftool
  • [PATCH net-next v4 1/3] tools: rename tools/net directory to tools/bpf
  • [PATCH net-next v4 2/3] tools: bpf: add bpftool
  • [PATCH net-next v4 3/3] tools: bpftool: add documentation
• Eric Dumazet, [PATCH net] socket, bpf: fix possible use after free
• Stephen Hemminger, [PATCH net-next] samples/bpf: fix warnings in xdp_monitor_user
• Martin KaFai Lau, [PATCH net-next] bpf: Fix compiler warning on info.map_ids for 32bit platform
• Yonghong Song, [llvm] r314469 - bpf: fix a bug for disassembling ld_pseudo inst
• Paul Chaignon, [PATCH RFC v3] bpf: allow map helpers access to map values directly
• Mark Rutland, EBPF-triggered WARNING at mm/percpu.c:1361 in v4-14-rc2
• Yonghong Song, [llvm] r314376 - bpf: add new insns for bswap_to_le and negation
• Martin KaFai Lau, [PATCH net-next 0/5] bpf: Extend bpf_{prog,map}_info
  • [PATCH net-next 1/5] bpf: Add name, load_time, uid and map_ids to bpf_prog_info
  • [PATCH net-next 2/5] bpf: Add map_name to bpf_map_info
  • [PATCH net-next 3/5] bpf: libbpf: Provide basic API support to specify BPF obj name
  • [PATCH net-next 4/5] bpf: Swap the order of checking prog_info and map_info
  • [PATCH net-next 5/5] bpf: Test new fields in bpf_attr and bpf_{prog,map}_info
• Matt Redfearn, [PATCH] MIPS: bpf: Fix uninitialised target compiler error
• Edward Cree, [PATCH v2 net-next 0/2] bpf/verifier: disassembly improvements
  • [PATCH v2 net-next 1/2] bpf/verifier: improve disassembly of BPF_END instructions
  • [PATCH v2 net-next 2/2] bpf/verifier: improve disassembly of BPF_NEG instructions
• Daniel Borkmann, [PATCH net-next 0/6] BPF metadata for direct access
  • [PATCH net-next 1/6] bpf: rename bpf_compute_data_end into bpf_compute_data_pointers
  • [PATCH net-next 2/6] bpf: add meta pointer for direct access
  • [PATCH net-next 3/6] bpf: update bpf.h uapi header for tools
  • [PATCH net-next 4/6] bpf: improve selftests and add tests for meta pointer
  • [PATCH net-next 5/6] bpf, nfp: add meta data support
  • [PATCH net-next 6/6] bpf, ixgbe: add meta data support
• Jason Wang, [PATCH net-next] virtio-net: correctly set xdp_xmit for mergeable buffer
• Yonghong Song, [PATCH] bpf: add support for neg insn and change format of bswap insn
• Edward Cree, [PATCH net-next] bpf/verifier: improve disassembly of BPF_END instructions
• Cong Wang, [Patch net-next v2] net_sched: use idr to allocate bpf filter handles
• Jiong Wang, [llvm] r313958 - bpf: refactor inst patterns with more mnemonics
• Jiong Wang, [llvm] r313959 - bpf: refactor inst patterns with better inheritance
• Jiong Wang, [llvm] r313960 - bpf: add 32bit register set
• Jiong Wang, [llvm] r313961 - bpf: initial 32-bit ALU encoding support in assembler
• Craig Gallek, [PATCH net-next v2] bpf: Optimize lpm trie delete
• Joel Fernandes, [PATCH v4 0/4] Add cross-compilation support to eBPF samples
  • [PATCH v4 1/4] samples/bpf: Use getppid instead of getpgrp for array map stress
  • [PATCH v4 2/4] samples/bpf: Enable cross compiler support
  • [PATCH v4 3/4] samples/bpf: Fix pt_regs issues when cross-compiling
  • [PATCH v4 4/4] samples/bpf: Add documentation on cross compilation
• Daniel Borkmann, [PATCH iproute2 master 0/2] BPF/XDP json follow-up
  • [PATCH iproute2 master 1/2] json: move json printer to common library
  • [PATCH iproute2 master 2/2] bpf: properly output json for xdp
• Edward Cree, [PATCH net] net: change skb->mac_header when Generic XDP calls adjust_head
• Jason Wang [PATCH net-next 1/3] virtio-net: remove unnecessary parameter of virtnet_xdp_xmit()
  • [PATCH net-next 2/3] virtio-net: add packet len average only when needed during XDP
  • [PATCH net-next 3/3] virtio-net: support XDP_REDIRECT
• Eric Dumazet, [PATCH net] bpf: do not disable/enable BH in bpf_map_free_id()
• Jiong Wang, [PATCH RFC 0/4] Initial 32-bit eBPF encoding support
  • [PATCH RFC 1/4] Improve instruction encoding descriptions
  • [PATCH RFC 2/4] Improve class inheritance in instruction patterns
  • [PATCH RFC 4/4] Initial 32-bit ALU encoding support in assembler
  • [PATCH RFC 3/4] New 32-bit register set
• Yonghong Song, [PATCH net] bpf: one perf event close won't free bpf program attached by another perf event
• Daniel Borkmann, [PATCH net v2] bpf: fix ri->map_owner pointer on bpf_prog_realloc
• Yonghong Song, [llvm] r313593 - bpf: add inline-asm support
• Craig Gallek, [PATCH net-next 0/3] Implement delete for BPF LPM trie
  • [PATCH net-next 1/3] bpf: Implement map_delete_elem for BPF_MAP_TYPE_LPM_TRIE
  • [PATCH net-next 2/3] bpf: Add uniqueness invariant to trivial lpm test implementation
  • [PATCH net-next 3/3] bpf: Test deletion in BPF_MAP_TYPE_LPM_TRIE
• Tobias Klauser, [PATCH] bpf: devmap: pass on return value of bpf_map_precharge_memlock
• Simon Dardis, [llvm] r313281 - [bpf] Fix test to always use little endian.
• Edward Cree, [PATCH net] bpf/verifier: reject BPF_ALU64|BPF_END

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.