# Keynote: Why decentralised open source cloud infrastructure matters! The security and the personal data on the internet are under attack by hackers and international espionage programs. The question who controls what data and application becomes more important every day. And governments care more and more about Digital Sovereignty and how this effects the future of countries. The GDPR was supposed to give people more control over their data. But now that the European Court of Justice declared the US Cloud Act incompatible with the GDPR the question arises how data should be stored and processed in the future. The Nextcloud community is building an open source and fully federated and distributed network for files and communication. Everyone can run a Nextcloud server at home or somewhere on the internet and collaborate and share with everyone else. Nextcloud can be used to provide file access, syncing, sharing, calendar, contacts, video calling, music and video streaming in a distributed way. This talk will cover the current problems with surveillance and espionage and strategies on how to fix this problem. It will also discuss the current and upcoming federation features of Nextcloud and how to become a part of the community. # Keynote: Ctrl-Alt-Del Learning to Love Legacy Code The world runs on legacy code. For every greenfield progressive web app with 100% test coverage, there are literally hundreds of archaic line-of-business applications running in production - systems with no tests, no documentation, built using out-of-date tools, languages and platforms. It’s the code developers love to hate - it’s not exciting, it’s not shiny, and it won’t look good on your CV - but the world runs on legacy code, and, as developers, if we’re going to work on anything that actually matters, we’re going to end up dealing with legacy. To work effectively with this kind of system, we need to answer some fundamental questions. Why was it built this way in the first place? What’s happened over the years it’s been running in production? And, most importantly, how can we develop our understanding of legacy codebases to the point where we’re confident that we can add features, fix bugs and improve performance without making things worse? # Reviving and evaluating Thompson's backdoor in OpenBSD's `make`. 38 years ago Ken Thompson presented¹ a well-known, binary-only backdoor based on two capabilities of self-hosted compilers: self-replication and their ability to learn. Other-than-compilers system components are self-hosted, like OpenBSD's `make`: in order to build a new `make` binary from source, you need an existing `make` executable and this one may have been _educated_ to misbehave! This talk covers the technical implementation of such backdoor and tries to evaluate trust-improving techniques for self-hosted components. Samuel AUBERTIN is a PhD student in the Systems & Software Security at EURECOM² and cybersecurity consultant for IBM in France. 1. [Reflections on Trusting Trust - ACM Turing award lecture](https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf) 2. [S3@EURECOM](https://www.s3.eurecom.fr/) # NetBSD 10: 3 Years in the Making NetBSD 10 has been in development now for almost three years. In this period, there's been many amazing things, pain, and tears happening. How has this affected the NetBSD community and what can we look to in the future? NetBSD 10 has been in development now for almost three years, since August 2019. In this period, there's been many amazing things happening. Changes have affected almost every area of the operating system, making NetBSD 10 a landmark release. NetBSD 10 will boast vastly superior performance, hardware support, and a host of new features. However, some events during the development process have been very controversial, and there have been periods where users were warned about a very unstable CURRENT. Nia discusses how some of the significant changes in NetBSD 10 have affected the NetBSD community, and the NetBSD's developers' approach to resolving conflicts and ensuring a stable release. # Architectures vs the ports tree: a losing battle ? Getting 3rd party software to run on lots of architectures is complicated. This talk will cover most strategies currently used to cope within the OpenBSD ports tree, ranging from specific port issues, like arch-specific atomic operations or intel CPU variations to generic naming/building dependency issues to figure out what to build on a given architecture and to build a maximum number of ports on exotic architectures. Over the years, we've experienced it all, unfortunately. These days, we got guidelines for upstream for dealing with actual binary package creation, as well as tools within the framework related to declaring specific architectures needs, and dealing with somewhat lacking dependencies. There is also the deeper problem of software needing more and more resources to build/run (memory mostly, though cpu time can be an issue) and the necessity of coping with that gracefully, while acknowledging the slow death of less-abled architectures. # Steps Towards Faster OpenBSD Networking For a long time OpenBSD network stack was not running on multiple processors in parallel. Partially it has been made MP ready in the previous releases. This talk gives an overview of what has been achieved, explains pitfalls, and shows the tooling used. In kernel development, and especially in multi processor environments, a good test equipment is necessary. The test results provided on http://bluhm.genua.de/perform/results/perform.html give insight about progress during the development process. Interesting changes and their effects will be explained. # Making FreeBSD QUIC The Internet has continued to evolve throughout its life and with these changes the protocols we use to move information around have had to adapt to keep up. TCP, the work horse of the Internet has reliably been delivering bytes for many decades, but the protocol lacks room to grow to meet future challenges. QUIC was created as a new protocol to enable future evolution in how we move bits around the Internet. QUIC adds always Authentication and Encryption, features that allow much improved congestion control, better features for CDN operators and higher reliability for end users. Once QUIC escaped into the IETF it followed an open development process, but other than Mac OS laptops for development, QUIC was mostly designed and tested on Linux systems. With this Linux focus we have to wonder if BSD platforms are being left behind? This talk looks at the performance of QUIC implementations on Linux and FreeBSD and explores the changes that made QUIC implementations faster. We will will dig into network performance and techniques that can used to understand the performance of any application on FreeBSD. With this we try to answer the question, How can we make FreeBSD QUIC? # Taming the fox ### Adding sandboxing to Firefox In addition to all its security features/techniques used in the base system (listed at https://www.openbsd.org/innovations.html), OpenBSD provides its own 'sandboxing' mechanisms through pledgde() (a process pledges it only needs a subset of syscalls) and unveil() (a process declares it only needs to access a specific list of directories/files, with specific modes) While those APIs are simple to use in the OpenBSD base system where all code is layed out/managed according to the coding style of the project, we'll see how they can be used in the context of a multi-process monster with more lines of code than a full fledged operating system: Firefox. Firefox supports sandboxing on its mainstream platforms since a while, and with the Electrolysis and Fission projects, the browser has been moving to a many-processes architecture, while still dealing with a large codebase that is constantly evolving. We'll see that integrating pledge()/unveil() in this architecture has not been a matter of LOC, but rather many hours of debugging, for a result that is usable (eg enabled by default, like all security features in OpenBSD) but improvable. # So you want to add a system call? Adding a new system call is notionally simple, but there are numerous edge cases that can confuse even senior developers. In this talk I cover the process of adding a system call including special handing for ABI compatibility layers like FREEBSD32. I will also cover the extra requirements to support upcoming CHERI-extended architectures. # Block Cloning in OpenZFS OpenZFS file system uses Copy-on-Write method to store data. This allows to quickly snapshot and clone ZFS datasets. In theory this should also allow for easy file cloning, but it was never implemented. Until now. This talk will present my implementation of block cloning for OpenZFS, that allows to quickly clone files (thus saving space), move files between datasets and restore files from snapshots without additional space requirements. # FreeBSD audio in amateur and professional setups FreeBSD has astonishingly good support for audio that is rarely talked about Although FreeBSD was not created with professional audio support in mind, over time it became one of the best alternatives out there. In recent years this operating system received numerous improvements which make it as good as it is today. This presentation will be about audio advancements made in those years, what was improved and what is left to do to make FreeBSD event better solution for audio studios. Presenter will demonstrate his own studio from the view of FreeBSD and talk about different ways of improving audio performance. Special care is taken on how not to need specialized hardware or PC intended just for audio. # GPU passthrough with bhyve ### Update of the current state of GPU passthrough for bhyve VMs This presentation is a follow up of my presentation at the FreeBSD Vendor Summit in 2021. It focuses on the current state of GPU passthrough with a bhyve VM. FreeBSD includes the bhyve hypervisor to create virtual machines. In order to increase the performance of a VM, bhyve supports PCI passthrough. When using PCI passthrough, a PCI device is exclusively assigned to a VM. Even the FreeBSD system has no access to the PCI device. Therefore, the VM can directly access the PCI device without any emulation. The support for PCI passthrough doesn't include GPU devices because they require some special handling. We are actively working on a solution to add GPU passthrough support to bhyve. In 2021 many GPUs were supported in an off-tree branch. In the meanwhile many changes regarding GPU passthrough are merged into upstream bhyve. For that reason, it's supported on some hardware and some guests now. The presentation will talk about which hardware and guests are supported on upstream bhyve, how GPU passthrough can be used and what has to be done to support more hardware and guests. # How OpenBSD transforms a bag of blocks into a set of useful filesystems Making disks useful An in-depth exploration of how OpenBSD uses fdisk(8), disklabel(8), installboot(8) and its installation scripts to take a block device and create a set of usable filesystems which is optionally a bootable environment. The emphasis will be on recent work involving GPT, UEFI and overcoming Arm, Arm64, Riscv64, Powerpc64 and Apple Silicon obstacles. # devsetup: local development environment based on pkgsrc Package once, deploy everywhere pkgsrc is the official source of software packages for NetBSD and SmartOS; that much is clear. It has the focus of developers and users on these two platforms, and it is expected to work well there. In this model, pkgsrc is typically used in _privileged_ mode (the default) where the packages are installed and managed by the system administrator: users and groups are created for packages as required, additional services are provided and can be registered with the Operating System. Another selling point for pkgsrc is its portability, since it is available and also working well on a number of other platforms: Linux distributions, FreeBSD (and derivatives like DragonFlyBSD), OpenBSD, or macOS are also supported. But on all of the platforms mentioned, pkgsrc can now also work just as well in _unprivileged_ mode. In this mode, any local user with a C compiler can clone pkgsrc, build packages, and manage these packages themselves, by themselves, for themselves. Thanks to the introduction of the `SYSCONFBASE` setting in `mk.conf`, together with ongoing work on the RC scripts provided, local users can even build, install, run, and manage services entirely through pkgsrc's packages. This brings a new use case for pkgsrc across every platform supported, old or modern: complete development environments can be based on pkgsrc releases, and work and behave consistently regardless of the local platform. This principle is illustrated in a new tool, called _devsetup_, where a single script can be used to bootstrap and maintain a complete development environment, thanks to pkgsrc. With devsetup, it only takes a single command to reproduce instances of pkgsrc across systems and platforms. This is ideal for pkgsrc developers and package maintainers, when tracking bugs or improving portability on unfamiliar platforms. This is particularly true on macOS for instance, where it can conveniently replace solutions like Homebrew. This presentation will focus on pkgsrc's unprivileged mode, from bootstrapping to package management. It will dig deeper into the internals of the `pkgtools/rc.subr` support system for the services provided by pkgsrc, and explain the need for the additional `SYSCONFBASE` setting. From there, ongoing and necessary changes to the RC scripts provided will be discussed, considering the differing contexts and environments in which services may be needed and used. # The "other" FreeBSD optimizations used by Netflix to serve video at 800Gb/s from a single server. ### Quantifying the importance of several FreeBSD optimizations to serving web traffic at extreme speeds. In previous talks, I've focused on new optimizations such as NUMA and NIC kTLS offload, which are critical to serve 800Gb/s. In this talk, I'll focus on the contributions of older, but equally critical optimizations, such as - TCP segmentation offload (TSO) - TCP large receive offload (LRO) - VM optimizations such as the UMA VM pg cache zone and batchqueues - Asynchronous sendfile - Unmapped (extpg) mbufs I will briefly describe the importance of each of these unsung optimizations, explain why they help, and quantify the value of each optimization using a metric I call percent-CPU per Gb. I will wrap up by putting them all together and showing results from an experimental Netflix server serving production traffic at 720Gb/s. # Scaling ZFS for NVMe ### Removing Bottlenecks Learn about how ZFS is being adapted to the ways the rules of storage are being changed by NVMe. In the past, storage was slow relative to the CPU so requests were preprocessed, sorted, and coalesced to improve performance. Modern NVMe is so low latency that we must avoid as much of this preprocessing as possible to maintain the performance this new storage paradigm has to offer. An overview of the work Klara has done to improve performance of multiple ZFS pools of large numbers of NVMe disks on high thread count machines. - Fixing the scaling of the TaskQueue interface - Removing bottlenecks to synchronous writes - Reducing contention on the TaskQueues - Reducing contention on the vdev queue lock A walkthrough of how we improved performance from 3 GB/sec to over 7 GB/sec of writes. # FreeBSD containers in production. A NSFW guide. An overview and summarization of running FreeBSD containers on the cloud in production. Two years ago, I joined a company to design and implement the entire backend stack, in which we eventually chose to use FreeBSD to run a number of our backend services, packaged as containers, alongside other services running on Kubernetes. This talk will explain the decision's rationale and summarize the good, the bad, and the ugly of running FreeBSD containers on the cloud. I will also demonstrate the tools we (ab)used to create, orchestrate and distribute these services, evaluate the impact on DevOps and other aspects, and contrast them with the Linux counterparts. Last but not least, we will try to conclude if this is for everyone. # Puffy presents video conferencing (jitsi) Video conferencing has become common place nowadays and this presentation will show how to install, configure, operate, scale and secure a videoconferencing service with `jitsi` running on OpenBSD. The presentation will cover all bits and bolts to fully understand the components at play, their intercommunications and how this knowledge can be used to create a jitsi-on-OpenBSD setup that features a restricted (compartmentalized) setup using dedicated machines or -as presented- VMM based VMs, where each VM runs only one of the components. It'll be shown what's necessary to create a sensible pf.conf on each VM and how to add reverse proxy (relayd, haproxy) for distribution of workload. Also covering pitfalls/hints along underlying components as Java-JRE, IPv6 dualstack rammifications and what to lookout for on the client/browser side for interopability. # How far a naive FreeBSD container implementation can go ### Success and failures for a project with good intensions In this talk, I'm going to present the current status of `pot`, a framework to provide container-alike workflow on FreeBSD. `pot` started in 2018 and it has been developed mainly in shell scripts (the naive part of the title) to prove that FreeBSD is ready to support containers. While speaking about the current status, I'll talk about the last 2 years development, the challenges to make the framework "production ready" and what is still missing. I'll also present all the related projects, like Nomad support and Pothub, a community driven public registry for `pot` images. Additionally, I'm going to present some personal opinion about containers in FreeBSD, based on my experience in the project. # On FreeBSD, compliance, and getting up from under the bus ### The Blame Game continues (or: Squid Game for sysadmins) The financial and payment industries are dominated by dinosaurs, and those are not known for moving fast. The associated security compliance business is much the same. It should come as no surprise that we are still - three years after my 2019 talk on _FreeBSD and the absurdities of security compliance_ - regularly finding ourselves in puzzling and sometimes nonsensical situations where compliance, security and sanity are a "pick any two" proposition. That said: The side effects of dinosaurs trying to turn around too quickly can be quite painful if you're too close. In 2021, we got the rough end of the deal when the dinosaurs got in the way of some of the seedier elements on the Internet, who decided to take it out on us and others like us through a 9-month long wave of denial-of-service attacks. We've spent the better part of 20 years building a FreeBSD-based platform - routers, firewalls, servers and all. It was all chugging along nicely until someone decided to paint a big target on our backs and then turn away. The Blame Game just got real. What we've put FreeBSD through over the last year and a half is unlike anything we've seen in the previous 20. This is both a story of "look what we can do with BSD" and "look what we've put BSD through". # The FreeBSD build option, OpenZFS, bhyve, compat_linux, and jail.conf.d nexus A powerful whole that is far greater than the sum of its parts FreeBSD 13.1 marks a milestone that was decades in the making: The unification of functioning build options that control operating system features, the cross-platform OpenZFS file system and volume manager, the configuration file format for the bhyve hypervisor, a Linux compatibility layer, and the Jail container subsystem with jail.conf.d which can efficiently enhance the security of them. No other open source or proprietary operating system can match these features and this talk will describe how FreeBSD 13.1 onward can provide the platform for flexible and secure compute environments that embrace a "learn once, deploy for decades" philosophy. Available features of this stack include: Host and Container OS upgrades via OpenZFS snapshots Institutionally-contained Virtual Machines Lightweight and full-featured FreeBSD Virtual Machines Lightweight and full-featured FreeBSD Containers Lightweight and full-featured Linux and other OS Virtual Machines Lightweight and full-featured Linux Containers Virtual Machine management using popular Jail management tools All using in-base or near-base tools. This talk is aimed at new and moderately-experienced system operators thanks to its leveraging of in-based tools and documentation. # FreeBSD on OpenPOWER This presentation explains how to run FreeBSD on OpenPOWER based machines, we will shows how to install and run as bare metal installation and as a virtual machine. We will demo the automated installation of a lowest entry server machines, and virtual machines. We will also go over the efforts of the OpenPOWER Foundation to make powerpc64le a tier1 and our efforts to build smaller machine types. # FreeBSD coming from a Linux user This presentation is about a long time Linux user, using FreeBSD and will highlight some of the personal differences between Linux and BSD, including using the native tools on either, but also about common tooling. We will also go over Ansible, scripting and automation used to get the best result. This presentation is a personal experience presentation and is not a complete explanation, nor the best solution. # What is new in sudo and syslog-ng ### Alert on sudo events using syslog-ng FreeBSD is one of the most popular platforms to run syslog-ng. This summer, we expect a major new release of syslog-ng. It will feature data typing, which comes handy if you process logs inside syslog-ng or store logs to Elasticsearch or others. In his talk, Peter will explain the latest syslog-ng featues, and also the very latest advanced sudo features. The two topics are closely related, as many of the new sudo features are logging-related. JSON formatted logs are now available, sub-commands can be logged, and chroot is directly supported within sudo. All of these logs are automagically parsed by syslog-ng, which makes alerting within syslog-ng or storing to NoSQL databases or cloud services easier. # Recent progress in and around LibreSSL More than eight years after the fork, active development on LibreSSL continues. Feature and API additions happen on an as-needed basis and code is continually cleaned up, improved, rewritten, and removed. This talk will survey some of the major contributions to LibreSSL since the last LibreSSL talk at BSDCan 2019. These include Bob Beck's X.509 validator, Joel Sing's rewrite of the legacy TLS record layer and the major refactoring of the templated ASN.1 decoder, Ingo Schwarze's documentation improvements and the author's work on API compatibility with OpenSSL and several other features in libcrypto, libssl and libtls. Quite a bit of effort also went into porting and fixing OpenSSL's RFC 3779 support. This is code to handle X.509 extensions containing lists of IP addresses and AS numbers as used in the Resource Public Key Infrastructure (RPKI) and is used notably in OpenBSD's rpki-client(8). After a look into the past, there will also be a discussion of ongoing work, current goals and non-goals as well as pain points and upcoming challenges. # Memory barriers for BSD hackers In the good old days of sequential machines, memory loads and stores appeared to happen in the order you asked for. But today, compilers and multi-core CPUs conspire to wreak havoc on your memory operations. How do we restore order in this chaos? This talk will present modern software abstractions for managing memory ordering and how to think about it in real-world programs. This talk is aimed at an audience of hackers, not academics -- there will be no overwhelmingly detailed formal models. # How I learned to stop worrying and yank the USB There's a whole lot of device driver code out there that works so long as the device stays plugged in. But some devices, such as USB devices, can be removed while in use. This is a common cause of crashes on BSD systems. How can we make the operating system gracefully handle device removal? This talk will discuss problems and race conditions we found in the NetBSD driver framework, how we've adapted the APIs to address them, and strategies for making device drivers robust to devices suddenly disappearing. # Measuring Performance Overhead of Tracing with DTrace and eBPF Tools such as DTrace and eBPF allow performance engineers to measure the performance of almost any part of an operating system. These tools can be easily hooked into an arbitrary part of the code base and gather the exact statistics one needs. But then comes the question of the overhead. Even though tracing tools are often advertised as having almost no overhead, they still impose a penalty on the running system. The presentation focuses on designing and conducting benchmarks in order to measure the overhead of DTrace on FreeBSD and eBPF on Linux. We will compare a number of workloads and analyze the performance hit, which an operating system takes when being actively traced. The goal of the presentation is to see what kind of overhead one can expect when tracing a production system. # Virtualized Network Infrastrukture for VM Clusters When running VM environments one does not always have the luxury to control all parts in a way one might dream of. Since it is quite easy to build a highly available routing & edge firewall setup with FreeBSD using technologies like carp, pf and pfsync with relatively cheap and easily available hardware, this requires you to be able to deploy and manage this hardware which might not be possible in some situations. In this talk I want to discuss in how you can still deploy a similar setup using VMs and what additional challenges you will face. To help judging if planing/running such a setup is actually worth you time in a situation you might face some challenges this talk will go through things like * interesting idiocies in virtual machine networking & drivers * how to prepare for the day when something is broken and now you can't access * the webinterface to get to the console of your virtual router since this * would need the routing to be up and running * how is the performance impacted by this? # nsh Network Shell on top of OpenBSD made for network administrators nsh is a tool that allows a network administrator harness the power of OpenBSD and the tools that are present in the OpenBSD base operating system. nsh is an an alternative or supplement: ifconfig(8) sysctl(8) route(8) and encapsulates configuration for other daemons into one place, with its own command language similar to the configuration language used by many network appliance vendors. Effectively nsh + OpenBSD can allow an un-inducted network administrator harness the true power of OpenBSD for just about any network role or tasks. from meat and potatoes routing to a layer7 TLS terminating, load balancing and stateful firewall failover appliance. In this presentation the speakers will demo nsh and how it integrates with a base OpenBSD install. The speakers will discuss the current nsh design and features, and what lies in the future for nsh. # Writing Custom Command in FreeBSD's DDB Kernel Debugger FreeBSD's kernel includes an in-kernel debugger (DDB) inherited from 4BSD. DDB's command interface provides limited capabilities for examining data, but new commands can be added relatively easily to add pretty-printing of data structures. This talk goes over the basic APIs available to DDB command handlers and how to register handlers as new commands. It will also cover more advanced topics such as manually parsing DDB commands to support richer command arguments.