diff --git a/content/about/banner.jpg b/content/about/banner.jpg new file mode 100644 index 0000000..faf8037 --- /dev/null +++ b/content/about/banner.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cba0889e15ce6b6702e01944777f154ee6a51a981e20e325de7fbcc27b8663ee +size 1160689 diff --git a/content/about/index.md b/content/about/index.md new file mode 100644 index 0000000..0191a4b --- /dev/null +++ b/content/about/index.md @@ -0,0 +1,18 @@ ++++ +title = "About" +date = 2025-09-11T15:47:00Z +draft = false +summary = "Hi, I'm Nick! I'm a Software Developer." + +[hero] +src = "banner.jpg" +caption = 'Photo by bert b via Unsplash' ++++ + +Hi, I'm Nick! I'm a Software Developer at SAS Institute, and a recent graduate with First-Class Honors in M.Eng. Computer Science from the University of Manchester. + +I enjoy working on open-source projects and exploring DevOps practices, secure hosting, and cloud technologies in my homelab. My interest in fintech keeps me experimenting with modular programming and data integration for real-time analysis. Outside of work, you'll often find me on photo walks, cooking, hiking -- that is, when the weather allows it, or hunting for hidden music gems, proudly landing in the top 1% on Spotify for most obscure tastes. + +Feel free to check out my [LinkedIn](https://www.linkedin.com/in/nikolaos-karaolidis/), or reach out [directly](https://social.karaolidis.com/) if you'd like to connect! + +Views expressed on this page and on my social media are my own and do not reflect those of my employer. diff --git a/content/posts/_index.md b/content/posts/_index.md new file mode 100644 index 0000000..13c2f42 --- /dev/null +++ b/content/posts/_index.md @@ -0,0 +1,4 @@ ++++ +title = "Posts" +details = false ++++ diff --git a/content/posts/custom-linux-kernel/hero.jpg b/content/posts/custom-linux-kernel/hero.jpg new file mode 100644 index 0000000..9dd7a48 --- /dev/null +++ b/content/posts/custom-linux-kernel/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5e383ccd732ada470914bd3e754a58a65f89428a14c779f9d92db5f9efa9a942 +size 2730065 diff --git a/content/posts/custom-linux-kernel/index.md b/content/posts/custom-linux-kernel/index.md new file mode 100644 index 0000000..8f6b3c0 --- /dev/null +++ b/content/posts/custom-linux-kernel/index.md @@ -0,0 +1,193 @@ ++++ +title = "Compiling a Custom Arch Linux Kernel" +date = 2021-12-31T17:29:00Z +draft = false +summary = "Last week, I finally gave in and tried my luck by compiling a custom Linux kernel. After several days of work and a lot of hair-pulling..." +tags = ["Guide", "Linux"] + +aliases = ['/custom-linux-kernel'] + +[hero] +src = "hero.jpg" +caption = 'Photo by Yancy Min via Unsplash' ++++ + +> A Linux user, a vegan, and an atheist walk into a bar. I know because they told everyone. + +During these past few months, I have been switching between various kernels depending on the work I wanted to do. Sometimes I needed an [IOMMU patch](https://aur.archlinux.org/packages/linux-vfio/), some other times I wanted to try some Linux [gaming](https://github.com/zen-kernel/zen-kernel), and, more often than not, I was too slow to select the correct GRUB entry and ended up using the [default](https://github.com/archlinux/linux) Arch Linux kernel. + +Last week, I finally gave in and tried my luck by compiling a custom kernel tailored specifically for my [new laptop]({{% ref "/posts/lenovo-legion-7" %}}). After several days of work and a lot of hair-pulling, I managed to boot into a kernel which only had support for the exact hardware I needed. + +```bash +$ cat boot.txt +Startup finished in 4.793s (firmware) + 1.253s (loader) + 1.408s (kernel) + 3.794s (userspace) = 11.251s +graphical.target reached after 3.794s in userspace +$ cat boot-eirene.txt +Startup finished in 4.781s (firmware) + 1.264s (loader) + 1.138s (kernel) + 3.365s (userspace) = 10.551s +graphical.target reached after 3.365s in userspace +``` + +Besides the slightly improved boot times, I also saw a surprising increase in battery life (7.9 Watts used when idle compared to 11) and about 500MBs more free RAM. + +In this post, I will be explaining the main steps I followed when creating this kernel, hopefully helping anyone that finds themselves in a similar situation. You can find the finished configuration in this project's [Git repo](https://git.karaolidis.com/karaolidis/linux-eirene). + +## Preparation + +> [!warning] Disclaimer +> I am not an expert and I don't claim to be one. I'm just a CompSci student who enjoys tinkering with his system and trying to optimize it as much as possible while sacrificing my sanity in the process. I am not responsible if you break your system or lose important files. + +Before you begin, I recommend installing a normal Linux distribution with good support for your hardware and making sure that everything is working how it's supposed to. In my case, I used Arch Linux with the latest kernel. + +Once you confirm that everything is working, you should also store the output of some commands so that you have a working reference point when booting with your custom kernel: + +- `lspci -nnkkvvv` for verbose hardware and driver details +- `lsmod` for loaded kernel modules +- `lsusb` for connected devices on various USB busses +- `lscpu` for CPU information + +You should also have some clear goals before you start working on your kernel's configuration. I originally made the mistake of going in blind, which made me waste several hours debugging boot errors, changing previously-set options, and eventually forcing me to restart from scratch. + +### Installing build dependencies + +Once you are ready to start, install the packages you need for building the kernel. If you are using Arch, this can be easily done by running: + +```bash +$ pacman -S base-devel xmlto kmod inetutils bc libelf git cpio perl tar xz +``` + +### Acquiring the source code + +After installation is complete, go ahead and download the latest kernel source code. You can do that by either going to https://www.kernel.org/, grabbing the latest tarball, and unpacking it, or by using git and pulling from the official mirror: + +```bash +$ git clone https://github.com/torvalds/linux +``` + +You should also make sure that the kernel tree is clean by running `make mrproper` in the downloaded directory. + +## Configuration + +Now it's finally time to configure the kernel. What I recommend doing is getting your distribution's default `.config` by running `zcat /proc/config.gz > .config`, and [compiling]({{% ref "/posts/custom-linux-kernel#compilation" %}}) it to act as a sanity check for further `.config` changes. + +### Advanced configuration + +Once you've made sure that the default config is working, you can go ahead and run `make clean` to remove the previous build and `make nconfig` to open the configuration menu. + +The sheer number of choices that will appear in front of you will be overwhelming, but don't let that discourage you. If you use a slow and methodic approach you'll realize that while compiling a custom kernel is time-consuming, it is certainly not hard. + +Just a word of advice: Do not try and configure everything in one run. There is a high chance that you will break something and spend more time debugging it than if you were to make small, gradual changes, recompiling every once in a while to check if everything is working. + +When I was creating my config, I had 3 main goals in mind: + +- Keep it as minimal as possible +- Improve performance as much as possible without harming battery life +- Disable any logging / debugging capabilities that might increase runtime overhead + +I'm not going to go through every choice I made since most of the options are architecture-dependent and analyzing all use-cases would take months. Even if I did, if you copied my config without understanding the functionality behind each feature, you would most likely end up with an unbootable system. + +There are a couple of resources you can use to learn what each option does: + +- The in-built help text, available by pressing `h` +- [DOTSLASHLINUX](https://firasuke.github.io/DOTSLASHLINUX/post/the-linux-kernel-configuration-guide-part-1-introduction/)'s kernel configuration guide +- The [Gentoo Wiki](https://wiki.gentoo.org/wiki/Kernel/Gentoo_Kernel_Configuration_Guide) +- Good ol' Google + +The general strategy I followed when working on this project was to first get the easy things out of the way: These included options like CPU architecture support, compression, filesystems, etc. I then worked my way through one submenu at a time, recompiling the kernel to make sure I didn't break anything. I recommend leaving the behemoth that is the Device Drivers menu for last since that will take the most time and contains multiple options dependent on other settings. + +### Applying Patches + +Now would also be a good time to apply any needed patches. In my case, I wanted to apply an [ACS Override](https://aur.archlinux.org/cgit/aur.git/tree/?h=linux-vfio) patch so that I could have more control over my IOMMU Groups. You can easily apply .patch files by moving them to the kernel source directory and running: + +```bash +$ patch -Np1 < [name-of-patch] +``` + +If you don't get any rejects, you are good to go for compilation. + +## Compilation + +Now that you've made some changes to your config, it's time to compile it. Go ahead and run the following commands: + +```bash +$ make +$ make modules +$ sudo make modules_install +``` + +Each step should take a decent amount of time, depending on how large the kernel is. In my case, it took about 10 minutes on a Ryzen 7 5800H. If everything went well, you should notice a new `arch/x86/boot/bzImage` file. + +## Installation + +Now that you have finished compiling your kernel, it is finally time to install it. + +### Kernel + +Go ahead and run the following command. You can name the resulting file as you wish, provided that it is prefixed with `vmlinuz`. In my case, I named it `vmlinuz-linux-eirene`: + +```bash +$ cp -v arch/x86/boot/bzImage /boot/vmlinuz-linux-eirene +``` + +### Initial RAM disk + +Once you have copied your kernel to the `/boot` directory, you should also generate an initial RAM disk. You can easily do that by copying the default kernel's preset: + +```bash +$ cp /etc/mkinitcpio.d/linux.preset /etc/mkinitcpio.d/linux-eirene.preset +``` + +After that, edit the new file and make sure to change it so it matches the name you selected. + +```ini hl_lines=[1, 4, 9, 13] +# mkinitcpio preset file for the 'linux-eirene' package + +ALL_config="/etc/mkinitcpio.conf" +ALL_kver="/boot/vmlinuz-linux-eirene" + +PRESETS=('default' 'fallback') + +#default_config="/etc/mkinitcpio.conf" +default_image="/boot/initramfs-linux-eirene.img" +#default_options="" + +#fallback_config="/etc/mkinitcpio.conf" +fallback_image="/boot/initramfs-linux-eirene-fallback.img" +fallback_options="-S autodetect" +``` + +### Bootloader + +Finally, add an entry for your new kernel in your bootloader's configuration file. If you are using GRUB, you can simply run the following command: + +```bash +$ grub-mkconfig -o /boot/grub/grub.cfg +``` + +If you didn't get any errors, you should be able to select the newly-compiled kernel the next time you boot your computer! + +## Troubleshooting + +One easy way to check for errors is to look at your system's journal: + +```bash +$ journalctl -p 3 -b +Dec 31 17:44:50 eirene kernel: Spectre V2 : Spectre mitigation: kernel not compiled with retpoline; no mitigation available! +Dec 31 17:44:50 eirene kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.PB2], AE_NOT_FOUND (20210930/dswl> +Dec 31 17:44:50 eirene kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20210930/psobject-220) +Dec 31 17:44:52 eirene systemd-modules-load[318]: Failed to find module 'nvidia' +Dec 31 17:44:52 eirene systemd-modules-load[318]: Failed to find module 'nvidia-uvm' +Dec 31 17:44:52 eirene systemd-backlight[718]: Failed to get backlight or LED device 'backlight:acpi_video0': No such device +Dec 31 17:44:52 eirene systemd[1]: Failed to start Load/Save Screen Backlight Brightness of backlight:acpi_video0. +Dec 31 17:44:53 eirene kernel: Bluetooth: hci0: Failed to read codec capabilities (-56) +Dec 31 17:46:08 eirene lightdm[1491]: gkr-pam: unable to locate daemon control file +``` + +Keep in mind that not everything shown here is important, but it can be a good lead for finding the root cause of an issue. In the worst-case scenario, you can simply backtrack and try recently-changed config options one by one to see which one creates the problem. + +## Conclusion + +Even though configuring a kernel is challenging, the feeling of booting it for the first time and seeing the resulting performance/battery life increase is definitely satisfying. + +Maintaining that kernel is an entirely different story, but that can be fun as well, depending on how much you hate yourself, your free time, and your mental health :grinning:. + +Happy coding and happy new year! diff --git a/content/posts/lenovo-legion-7/hero.png b/content/posts/lenovo-legion-7/hero.png new file mode 100644 index 0000000..22ff5ab --- /dev/null +++ b/content/posts/lenovo-legion-7/hero.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c28fb4c50735f7b24df70b34c25e382d6b65101e02f39e46b8d89af2ca61143e +size 950870 diff --git a/content/posts/lenovo-legion-7/index.md b/content/posts/lenovo-legion-7/index.md new file mode 100644 index 0000000..87e1744 --- /dev/null +++ b/content/posts/lenovo-legion-7/index.md @@ -0,0 +1,107 @@ ++++ +title = "Lenovo Legion 7 (2021) and Arch Linux" +date = 2021-11-09T21:04:00Z +draft = false +summary = "The Lenovo Legion 7 is one of the best high-end laptops you can buy right now. Its sleek design, large screen-to-body ratio, and excellent cooling capabilities make it the perfect fit for anyone looking for a powerful machine in a decently portable package." +tags = ["Review", "Linux", "Guide"] + +aliases = ['/lenovo-legion-7'] + +[hero] +src = "hero.png" +caption = 'Photo by Hardware Canucks via YouTube' ++++ + +The Lenovo Legion 7 is one of the best high-end laptops you can buy right now. Although it's mainly aimed at a gaming audience, its sleek design, large screen-to-body ratio, and excellent cooling capabilities make it the perfect fit for anyone looking for a powerful machine in a decently portable package. + +The model I have is the Legion 7 16ACH-06 with the following specifications: + +- AMD Ryzen 7 5800H (8 Cores, 16 Threads, Base 3.2GHz, Turbo 4.4GHz) +- NVIDIA GeForce RTX 3070 Mobile +- GPU MUX Switch (Dedicated + Optimus Graphics) +- 32GB DDR4 3200MHz RAM +- 2TB M.2 NVMe PCIe SSD +- 16" QHD (2560x1600) 165Hz 3ms 500nits HDR400 IPS Display +- 300W Power Supply + +As with most laptops on the market, this device came pre-installed with Windows 10 Home as well as the Lenovo Vantage App which allows users to modify system settings typically accessed through the UEFI menu. + +## Linux Addiction + +Even though Windows is without a doubt the best operating system for gaming, that's not what I wanted to use this device for. As a CompSci student, I wanted to have a machine that allows me to code and compile projects on the go, manage multiple virtual machines, and have decent battery life. So, after checking that everything was working fine, I imaged the entire disk using [Clonezilla](https://clonezilla.org/) in case something went wrong, and proceeded to install Arch Linux. + +I'm not going to describe the entire installation process since the [Arch Wiki](https://wiki.archlinux.org/title/installation_guide) does it better than I could ever hope to, but I still want to mention a couple of things: + +1. I only made a single `ext4` partition (instead of separate root, home, and swap) and created a `/swapfile` for easy resizing later down the line. However, you can partition your drive however you see fit. +2. When running `pacstrap`, I recommend installing a lot more than just what the wiki recommends: + +```bash +$ pacstrap /mnt base linux linux-firmware amd-ucode efibootmgr grub git nano vim linux-headers net-tools networkmanager sudo +``` + +After installing a desktop environment / display manager / window manager combo (in my case, that's [LightDM](https://wiki.archlinux.org/title/LightDM) and [AwesomeWM](https://wiki.archlinux.org/title/awesome)), you should also install the necessary video drivers: + +```bash +$ pacman -S nvidia xf86-video-amdgpu +``` + +## Bug Fixing + +Most of the laptop's hardware was working out of the box, including: + +- Keyboard +- Touchpad +- RGB lighting presets (Fn+Space) +- Performance profiles (Fn+Q) +- Camera +- Microphone +- WiFi & Bluetooth +- I/O Ports +- GPU Switching (since you do that from the UEFI) +- Battery Conservation Mode (using a simple [script](https://git.karaolidis.com/karaolidis/legion-7/src/branch/master/bin/conservation-mode)) +- Multihead Support +- Suspend/Hibernate Support + +The default IOMMU groups were also decent, with the GPU Video and Audio buses being separate from the rest of the system. This, along with the GPU MUX switch shows promise for a single-GPU [PCI passthrough](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF) setup... + +Sadly, Linux is not officially supported by Lenovo on this model, so two major issues were immediately noticeable: backlight control and speaker audio. + +### Backlight Control + +> [!important] Update +> After updating to drivers `nvidia-510.xx`, brightness control no longer works in dedicated mode. For a temporary solution, try [downgrading](https://wiki.archlinux.org/title/downgrading_packages) back to `nvidia-495.46-9`. + +The backlight was a surprisingly easy fix. First, I had to add `nvidia.NVreg_RegistryDwords=EnableBrightnessControl=1` to my [kernel parameters](https://wiki.archlinux.org/title/kernel_parameters). After this, I created a couple of [udev rules](https://wiki.archlinux.org/title/Udev#udev_rule_example) to give non-root users access to the ACPI backlight controls. Finally, I wrote a couple of scripts that automatically get the current brightness level and add/subtract 10%. You can find all scripts as well as the udev rules in [this](https://git.karaolidis.com/karaolidis/legion-7) git repository. + +### Audio + +> [!important] Update +> As of kernel `5.18`, audio is no longer an issue. If you still can't get it to work, make sure that you have enabled `CONFIG_SERIAL_MULTI_INSTANTIATE` in your kernel config. If you need a guide on how to apply patches and build a custom kernel, you can check out [this]({{% ref "/posts/custom-linux-kernel" %}}) post. + +The audio issues are sadly almost impossible to fix. The 3.5mm headphone jack and Bluetooth audio work fine. However, the speakers have a couple of [Cirrus CS35L41](https://www.cirrus.com/products/cs35l41/) pre-amps that Linux doesn't recognize correctly, and therefore doesn't initialize. Despite that, there's still hope: [This](https://bugzilla.kernel.org/show_bug.cgi?id=208555) thread shows promising results with similar models and it is estimated that a working driver will be ready by [early 2022](https://bugzilla.kernel.org/show_bug.cgi?id=208555#c470). + +### Refresh Rate + +Another less important quirk is that Xorg prefers the 60Hz mode by default when using the dedicated GPU. This can easily be fixed by creating a `/usr/share/X11/xorg.conf.d/10-nvidia-refresh-rate.conf` file and adding the following contents: + +```xorg.conf +Section "Screen" + Identifier "Screen 0" + Subsection "Display" + Modes "2560x1600_165" + EndSubsection +EndSection +``` + +### Touchpad + +Finally, there is a small chance that the touchpad drivers won't load correctly when booting the laptop, forcing you to restart the machine. This can be fixed by creating a `modprobe` dependency: + +``` +softdep i2c_hid pre: pinctrl_amd +softdep usbhid pre: pinctrl_amd +``` + +## Conclusion + +Even though using Arch on this laptop is not 100% smooth sailing, I feel that the few issues it has will be ironed out during the next couple of months. So, if you are currently in the market for a powerful machine to install Linux on, I'd definitely recommend giving this one a shot - if you can find it in stock, that is. diff --git a/content/posts/nested-virtualization/game-over.png b/content/posts/nested-virtualization/game-over.png new file mode 100644 index 0000000..ae518a2 --- /dev/null +++ b/content/posts/nested-virtualization/game-over.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:264af4ac3870fead343df14389c61fa8040ab53ef8ac6f96a3a86e48d00732e1 +size 2195330 diff --git a/content/posts/nested-virtualization/hero.jpg b/content/posts/nested-virtualization/hero.jpg new file mode 100644 index 0000000..69bd2b7 --- /dev/null +++ b/content/posts/nested-virtualization/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dde88f53a6980a988a601486195e795d8cc9bcfe47dfee52a17b68cee47b36b6 +size 2928287 diff --git a/content/posts/nested-virtualization/host.png b/content/posts/nested-virtualization/host.png new file mode 100644 index 0000000..cbcc31b --- /dev/null +++ b/content/posts/nested-virtualization/host.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d3fde6c7f62ed505a704349848e9eb50da2a08cb4aaf980677d963579ee62085 +size 942396 diff --git a/content/posts/nested-virtualization/index.md b/content/posts/nested-virtualization/index.md new file mode 100644 index 0000000..2f67db0 --- /dev/null +++ b/content/posts/nested-virtualization/index.md @@ -0,0 +1,95 @@ ++++ +title = "Pushing Nested Virtualization to the Limits" +date = 2022-04-27T19:54:00Z +draft = false +summary = "Trying to install all versions of Windows inside nested VMs..." +tags = ["Experiment", "Virtualization", "Linux", "Windows"] + +aliases = ['/nested-virtualization'] + +[hero] +src = "hero.jpg" +caption = 'Photo by Didssph via Unsplash' ++++ + +I recently watched a video by SomeOrdinaryGamers in which he decided to install the first version of Windows and upgrade it all the way to Windows 10. + +{{< youtube id=KD3tuWy-z_w loading=lazy start=92 >}} + +Despite the video's thumbnail, I decided that it would a good idea to do something similar: Install all versions of Windows inside nested VMs. + +> [!warning] Disclaimer +> Do not try this at home. Or do, I don't really care. Just be warned that it is a massive waste of time :upside_down_face:. + +Sadly, I didn't have access to my supercomputer with 1000s of CPU cores and Terabytes of RAM at the time, so my Legion 7 would have to do. Its specs are as follows: + +- AMD Ryzen 7 5800H (8 Cores, 16 Threads, Base 3.2GHz, Turbo 4.4GHz) +- NVIDIA GeForce RTX 3070 Mobile +- 32GB DDR4 3200MHz RAM +- 2TB M.2 NVMe PCIe SSD + +About a month ago I completed a [single-GPU VFIO passthrough]({{% ref "/posts/vfio" %}}), which allowed me to have near-bare-metal performance on a Windows 10 VM running inside Arch Linux. This meant that my host was pretty much ready to go. + +![Host OS: Arch Linux](host.png) + +The plan was simple: Install virtualization software on every operating system and create the nested VMs. Easy, right? Well... The execution was a bit tougher. + +## First VM + +My first victim was my Windows 10 Gaming VM. If you want detailed instructions on how I created it, you can check out the post linked above. + +![First Virtual Machine: Windows 10](level-0.png) + +I originally tried going the Hyper-V route for my next VM but performance inside Windows 10 tanked significantly the moment I enabled it, forcing me to install VirtualBox. + +## Second VM + +After installing VirtualBox, I downloaded a [Windows 8.1 ISO](https://www.microsoft.com/en-us/software-download/windows8ISO) from Microsoft and created a new VM. + +![Second Virtual Machine Config](level-1-config.png) + +I gave this VM around half of the resources of Windows 10, making sure to also enable nested AMD-V. After a painfully nit-picky installation (thanks Microsoft), I was finally able to boot Windows 8.1. + +![Second Virtual Machine: Windows 8.1](level-1.png) + +General performance was OK-ish and the VM was definitely usable. However, networking was unfathomably slow, peaking at around 0.8MB/s. + +Luckily, VirtualBox provides easy folder sharing between hosts and guests, so all I had to do was download any needed files on Windows 10 and pass them through to Windows 8.1. + +After installing VirtualBox once again, it was time for the third VM. + +## Third VM + +This is where things started to get a bit annoying. As you might know, Microsoft has dropped support for Windows 7, so finding an ISO was a bit more difficult. Luckily, with a bit of google-fu, I found a website that hosted a direct download link. + +While configuring the VM, I noticed that I could no longer enable nested AMD-V. Uh-oh. + +![Third Virtual Machine Config](level-2-config.png) + +Staying optimistic, I pressed forward and started the Windows 7 installation. I quickly realized that this challenge was a lot harder than I originally expected. + +![Third Virtual Machine Config](level-2.png) + +The installation took a total of 4 painfully slow hours, ending with a blue screen of death right after the installer tried to reboot the VM. All hope was not yet lost though. + +I restarted the VM, and after a one-hour-long boot process (I am not joking), I was greeted with the Windows 7 desktop. Even though the VM booted, the user interface felt beyond sluggish, with every click taking several seconds to register, and animations tearing up the whole screen. + +However, this didn't stop me and my hubris. I installed VirtualBox again. + +## Fourth VM + +My next target was Windows XP. I deliberately skipped Vista because I was too scared of the performance impact it would have. After downloading a dodgy Windows XP ISO, I created a new VM inside VirtualBox. + +This is where I started to realize that my ~insanity~ experiment would soon come to an end. Almost all options, including multiple cores and hardware acceleration, were greyed-out. + +Despite that, I still tried to create a VM. After configuring the available options, I started the Windows XP installation. Sadly, after another hour, I was greeted with an error message saying that AMD-V was not available. + +![He's dead, Jim](game-over.png) + +## Conclusion + +In the end, I managed to create a measly 2 nested VMs before being stopped by VirtualBox limitations. + +I'm guessing that by using the more efficient KVM platform I would be able to nest at least 3, possibly 4 VMs. However, that is an experiment for another time. + +What did I learn from this experience? Absolutely nothing, other than that I am really good at finding ways to procrastinate. diff --git a/content/posts/nested-virtualization/level-0.png b/content/posts/nested-virtualization/level-0.png new file mode 100644 index 0000000..e458aab --- /dev/null +++ b/content/posts/nested-virtualization/level-0.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bcbaae9b640315031864d8802ebcd4c956fa1bafd8757c631d361a49c08cad65 +size 2043974 diff --git a/content/posts/nested-virtualization/level-1-config.png b/content/posts/nested-virtualization/level-1-config.png new file mode 100644 index 0000000..2277307 --- /dev/null +++ b/content/posts/nested-virtualization/level-1-config.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a0f452dcf957ce1f8be8db5fa164e0eb788cc9715570ebecdfae86097e6f07d2 +size 138277 diff --git a/content/posts/nested-virtualization/level-1.png b/content/posts/nested-virtualization/level-1.png new file mode 100644 index 0000000..be5758c --- /dev/null +++ b/content/posts/nested-virtualization/level-1.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:53f60611e2e61b5a9de90f6fb93e0841e342f08bd05b53d128eb8438f22aaf39 +size 2254762 diff --git a/content/posts/nested-virtualization/level-2-config.png b/content/posts/nested-virtualization/level-2-config.png new file mode 100644 index 0000000..5a7d52f --- /dev/null +++ b/content/posts/nested-virtualization/level-2-config.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1f474f9d820166ac86b41f0f73d111765f37e1cd4dd81838b3809edefbd53785 +size 70281 diff --git a/content/posts/nested-virtualization/level-2.png b/content/posts/nested-virtualization/level-2.png new file mode 100644 index 0000000..469ce7c --- /dev/null +++ b/content/posts/nested-virtualization/level-2.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:25e89dbed710611ad5927fbc8c8f51d14e1df9863d3a524e9417e227b73145c3 +size 2480016 diff --git a/content/posts/ultimate-home-server-part-1/hero.jpg b/content/posts/ultimate-home-server-part-1/hero.jpg new file mode 100644 index 0000000..630f8ae --- /dev/null +++ b/content/posts/ultimate-home-server-part-1/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e32228fb1549bb5475fdb1b1b78ffa7188109f2ff14943b84148891e32638786 +size 2222676 diff --git a/content/posts/ultimate-home-server-part-1/index.md b/content/posts/ultimate-home-server-part-1/index.md new file mode 100644 index 0000000..5b59f63 --- /dev/null +++ b/content/posts/ultimate-home-server-part-1/index.md @@ -0,0 +1,298 @@ ++++ +title = "Building the Ultimate Linux Home Server - Part 1: Intro, MergerFS, and SnapRAID" +date = 2021-07-24T10:59:00Z +draft = false +summary = "Turning an old desktop PC into a fully fledged Linux server using MergerFS and SnapRAID." +tags = ['Guide', 'Self-Hosting', 'Linux'] + +aliases = ['/ultimate-home-server-part-1'] + +[hero] +src = "hero.jpg" +caption = 'Photo by Florian Krumm via Unsplash' ++++ + +> [!important] Disclaimer +> This guide was written in 2021 and reflects my setup and recommendations at the time. Some tools, software versions, and best practices may have changed since then. Consider checking more recent resources alongside this post. + +A couple of months ago, while wasting time browsing Reddit, I discovered [r/homelab](https://www.reddit.com/r/homelab/). After looking at the top posts for a couple of weeks, I decided that I also wanted to join in on the action and build my own self-hosted Linux Server. + +I soon ran into a couple of issues. You see, as a university student without a full-time job, I really didn't have any disposable income to spend on expensive server racks, network equipment, and drive arrays. + +I did, however, have my old desktop which was collecting dust under my table after I had replaced it with a new laptop, so I decided to experiment and try to use that as my server. + +Some of the requirements I wanted this server to fulfil were the following: + +- The ability to combine and use drives of different sizes with a single mountpoint +- The ability to add new drives whenever space is running low without much hassle +- Protection against drive failure +- Automated backup of data and config files +- NAS array and backup solution for other devices +- Automated Movie, TV Show, Music, and Anime downloading, sorting, and tagging +- Personal streaming service for family and friends +- Automated torrent download client +- Personal cloud server accessible from anywhere +- Automatic updating of all services +- Host for any other projects/ideas/tools I wanted to use in the future +- Easy deployment of applications and services without worrying about dependencies/conflicts + +## Hardware + +The PC I used by no means contains server-grade hardware, but since I had originally built it as a gaming computer, it did the job well enough. Its specs are as follows: + +- **CPU**: Intel Core i7-6700K 4GHz Quad-Core +- **Cooler**: Cooler Master Seidon 120V +- **Motherboard**: ASRock Fatal1ty Z170 Gaming K4 +- **RAM**: Corsair Vengeance LPX 16 GB (2 x 8 GB) DDR4-2400 CL14 +- **GPU**: MSI Radeon R9 390 8GB +- **Case**: Phanteks Enthoo Pro ATX +- **Power Supply**: Cooler Master 750W 80+ Bronze Semi-modular ATX +- **Boot Drive**: Samsung 860 EVO 250GB SSD + +I could use the system as-is, but I decided to turn off the dedicated GPU in the BIOS since it didn't have any real purpose, and I didn't want to use as much power as a small city. + +As far as storage goes, I installed the operating system on the SSD and used a bunch of good-ol' spinning rust hard drives for everything else since most files would remain unchanged. + +## Operating System + +There is a lot of debate about which Linux distro is best to use for a server. Most people would recommend something like Ubuntu Server or Debian because of their stability, however, I decided to use Arch{{< sup "btw" >}} for a couple of really specific reasons: + +- I wanted a lightweight installation which I could tailor to my exact needs +- I wanted to become more proficient at using the Linux command line and its tools, instead of having everything get set up automatically + +Despite that, most of the things I will talk about will be the same or similar, independent of what distro you decide to use. + +The OS installation itself is beyond the scope of this article since there are lots of better-written guides that explain the process (The [Arch Wiki](https://wiki.archlinux.org/title/installation_guide), for example). + +## MergerFS + +There are definitely lots of different ways to go about setting up the storage system for your server. You could create a hardware RAID array, try out ZFS, or simply mount all your drives in your home directory and manage them manually. + +However, none of these solutions is nearly as flexible, inexpensive, and easy to use as [MergerFS](https://github.com/trapexit/mergerfs). + +> **Mergerfs** is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to **mhddfs**, **unionfs**, and **aufs**. + +In layman's terms, MergerFS allows you to combine drives of different sizes and speeds into a single mountpoint, automatically managing how files are stored in the background. + +``` +1TB + 2TB = 3TB +/disk1 /disk2 /merged +| | | ++-- /dir1 +-- /dir1 +-- /dir1 +| | | | | | +| +-- file1 | +-- file2 | +-- file1 +| | +-- file3 | +-- file2 ++-- /dir2 | | +-- file3 +| | +-- /dir3 | +| +-- file4 | +-- /dir2 +| +-- file5 | | ++-- file6 | +-- file4 + | + +-- /dir3 + | | + | +-- file5 + | + +-- file6 +``` + +Of course, there is a minor performance overhead when using this approach but as far as home servers are concerned, the advantages outweigh the disadvantages. + +### My Setup + +For my system I have 3 drives in total formatted as `ext4`, excluding the boot drive: + +- WD Blue 1TB 5400 RPM HDD - `/mnt/disk1` - Part of storage pool +- WD Red 4TB 5400 RPM HDD - `/mnt/disk2` - Part of storage pool +- WD RED 4TB 5400 RPM HDD - `/mnt/parity1` - SnapRAID parity, you must use your largest drive for this + +The three main storage drives are then *pooled* into a new directory called `/mnt/storage` where all my files can be accessed from. `/mnt/storage` contains the following subdirectories: + +- `/public`: Public folder accessible by all users +- `/public/media`: Media storage +- `/private`: Folder with personal user data for all users that store files on the server. +- `/configs`: Application config files + +### Installation + +Getting MergerFS up and running is pretty simple since you just need to install it using your preferred package manager, edit your `fstab` file, and reboot. + +```bash +$ yay -S mergerfs +``` + +After installing it, you need to find the disk IDs since the mapping of a device to a drive letter is not guaranteed to always be the same, even with the same hardware and configuration: `ls /dev/disk/by-id` + +Running this command will return something similar to this: + +```bash +$ ls /dev/disk/by-id +ata-Optiarc_DVD_RW_AD-5240S wwn-0x50014ee20d526ebe +ata-Samsung_SSD_860_EVO_250GB_S3YJNB0K512940F wwn-0x50014ee20d526ebe-part1 +ata-Samsung_SSD_860_EVO_250GB_S3YJNB0K512940F-part1 wwn-0x50014ee21329b6cf +ata-Samsung_SSD_860_EVO_250GB_S3YJNB0K512940F-part2 wwn-0x50014ee21329b6cf-part1 +ata-WDC_WD10EZRZ-00HTKB0_WD-WCC4J2AXKT3R wwn-0x50014ee2bcff024d +ata-WDC_WD10EZRZ-00HTKB0_WD-WCC4J2AXKT3R-part1 wwn-0x50014ee2bcff024d-part1 +ata-WDC_WD40EFAX-68JH4N0_WD-WX12D80N59SR wwn-0x5002538e403be893 +ata-WDC_WD40EFAX-68JH4N0_WD-WX12D80N59SR-part1 wwn-0x5002538e403be893-part1 +ata-WDC_WD40EFAX-68JH4N0_WD-WX52D104L73P wwn-0x5002538e403be893-part2 +ata-WDC_WD40EFAX-68JH4N0_WD-WX52D104L73P-part1 +``` + +What we are interested in are the lines containing the IDs of the partitions themselves instead of the entire drives (the `ata-xxx-part1` lines), so in this case: + +``` +ata-WDC_WD10EZRZ-00HTKB0_WD-WCC4J2AXKT3R-part1 +ata-WDC_WD40EFAX-68JH4N0_WD-WX12D80N59SR-part1 +ata-WDC_WD40EFAX-68JH4N0_WD-WX52D104L73P-part1 +``` + +Next, edit your `fstab` file, mount the partitions (including the parity drive) and create the MergerFS pool. + +``` +... +# hard drives +/dev/disk/by-id/ata-WDC_WD10EZRZ-00HTKB0_WD-WCC4J2AXKT3R-part1 /mnt/disk1 ext4 defaults 0 0 +/dev/disk/by-id/ata-WDC_WD40EFAX-68JH4N0_WD-WX52D104L73P-part1 /mnt/disk2 ext4 defaults 0 0 +/dev/disk/by-id/ata-WDC_WD40EFAX-68JH4N0_WD-WX12D80N59SR-part1 /mnt/parity1 ext4 defaults 0 0 + +# mergerfs +/mnt/disk* /mnt/storage fuse.mergerfs defaults,dropcacheonclose=true,allow_other,minfreespace=25G,fsname=mergerfs 0 0 +... +``` + +You can find a full list of options for your storage pool [here](https://github.com/trapexit/mergerfs#mount-options). + +After editing your `fstab` file, save and reboot. If everything went well, you should be able to create a file in `/mnt/storage` and see that the file was actually stored in one of the `/mnt/diskX` directories. + +## SnapRAID + +We have now finished setting up our storage pool, but what happens when one of our drives inevitably fails? This is where SnapRAID comes into play. Remember the parity drive we left unused until now? Well, this drive won't actually store any of your data, instead, it will hold parity information used to recover data if any disk dies. + +Keep in mind that using SnapRAID has a couple of caveats: + +- The parity drive must match or be larger than the size of the biggest data disk +- SnapRAID does not perform the parity "on write", meaning that you must manually invoke the `snapraid sync` command the recalculate the parity data (or use a tool like `snapraid-runner`). + +### Installation + + +```bash +$ yay -S snapraid +``` + +Next, create/edit your SnapRAID configuration file: + +```ini +# Defines the file to use as parity storage +# It must NOT be in a data disk +parity /mnt/parity1/snapraid.parity + +# Defines the files to use as content list +# You can use multiple specification to store more copies +# You must have least one copy for each parity file plus one. +# They can be in the disks used for data, parity or boot, +# but each file must be in a different disk. +content /var/snapraid.content +content /mnt/parity1/.snapraid.content +content /mnt/disk1/.snapraid.content +content /mnt/disk2/.snapraid.content + +# Defines the data disks to use +# The order is relevant for parity, do not change it +disk d1 /mnt/disk1 +disk d2 /mnt/disk2 + +# Excludes hidden files and directories (uncomment to enable). +#nohidden + +# Defines files and directories to exclude +# Remember that all the paths are relative at the mount points +# Format: "exclude FILE" +# Format: "exclude DIR/" +# Format: "exclude /PATH/FILE" +# Format: "exclude /PATH/DIR/" +exclude /lost+found/ + +# You might also want to exclude any log files or temporary DB files since these are changed frequently and might mess with the parity file. +exclude *wal +``` + +After editing `/etc/snapraid.conf`, try running `snapraid sync` as `root` to check if everything is configured correctly. Just keep in mind that this first sync could take a long time depending on the size of your drives. + +### Automation + +Since one of the requirements at the start of the article was automated backups, we are going to use [snapraid-runner](https://github.com/Chronial/snapraid-runner) to run a parity sync once every week: + +```bash +$ git clone https://github.com/Chronial/snapraid-runner.git /opt/snapraid-runner +``` + +After that create your configuration file (just make sure to fill in your email settings): + +```ini +[snapraid] +; path to the snapraid executable (e.g. /bin/snapraid) +executable = /usr/bin/snapraid +; path to the snapraid config to be used +config = /etc/snapraid.conf +; abort operation if there are more deletes than this, set to -1 to disable +deletethreshold = -1 +; if you want touch to be ran each time +touch = false + +[logging] +; logfile to write to, leave empty to disable +file = /var/log/snapraid.log +; maximum logfile size in KiB, leave empty for infinite +maxsize = 5000 + +[email] +; when to send an email, comma-separated list of [success, error] +sendon = success,error +; set to false to get full programm output via email +short = true +subject = [SnapRAID] Status Report +from = {fill in} +to = {fill in} +; maximum email size in KiB +maxsize = 500 + +[smtp] +host = {fill in} +; leave empty for default port +port = {fill in} +; set to "true" to activate +ssl = {fill in} +tls = {fill in} +user = {fill in} +password = {fill in} + +[scrub] +; set to true to run scrub after sync +enabled = true +percentage = 22 +older-than = 12 +``` + +We are then going to use [Cron](https://wiki.archlinux.org/title/Cron) to call snapraid-runner once every week, specifically at 12:00 every Sunday: `sudo crontab -e` + +```bash +... +0 12 * * 0 python /opt/snapraid-runner/snapraid-runner.py --conf /etc/snapraid-runner.conf +... +``` + +After saving the `crontab` file, SnapRAID will automatically back up your drives every week! + +## Final Thoughts + +Just by installing and configuring these two tools, we have managed to satisfy the first 4 requirements for our home server. We could stop right here and be good to go. However, there are a couple of things I strongly recommend doing before starting to host any services and exposing your server to the public: + +- Configure [SSH](https://wiki.archlinux.org/title/OpenSSH) and harden it using [2-factor authentication](https://wiki.archlinux.org/title/Google_Authenticator) +- Configure [fail2ban](https://wiki.archlinux.org/title/Fail2ban) to counter brute-force attacks +- Give your server a recognizable [hostname](https://wiki.archlinux.org/title/Network_configuration#Set_the_hostname) (in my case that's jupiter) +- Set up an SMTP client like [msmtp](https://wiki.archlinux.org/title/msmtp) so that your server can send you e-mail alerts +- Set up [S.M.A.R.T.](https://wiki.archlinux.org/title/S.M.A.R.T.) monitoring for your drives so that you get an early warning if one of your drives is about to fail. +- Remember the rule known as *Schrödinger's Backup*: The condition of any backup is unknown until a restore is attempted. Therefore I recommend setting up another [backup solution](https://wiki.archlinux.org/title/System_backup) other than SnapRAID, just in case. + +In the next part, we are going to be setting up Docker and Portainer for container management, Watchtower for automatic container updates, and OpenVPN for remote server management. diff --git a/content/posts/ultimate-home-server-part-2/docker.png b/content/posts/ultimate-home-server-part-2/docker.png new file mode 100644 index 0000000..27a8d6c --- /dev/null +++ b/content/posts/ultimate-home-server-part-2/docker.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c5330a16f53f3547d843e0e2efd1b42f96f4d0b6157ef929a643ab52f621e548 +size 71167 diff --git a/content/posts/ultimate-home-server-part-2/hero.jpg b/content/posts/ultimate-home-server-part-2/hero.jpg new file mode 100644 index 0000000..47c4f73 --- /dev/null +++ b/content/posts/ultimate-home-server-part-2/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:722a3d51441b023bc2b5536cd640a4596fc653da1f3b10d83f1f1a1420fb37c2 +size 3195467 diff --git a/content/posts/ultimate-home-server-part-2/index.md b/content/posts/ultimate-home-server-part-2/index.md new file mode 100644 index 0000000..6a5d732 --- /dev/null +++ b/content/posts/ultimate-home-server-part-2/index.md @@ -0,0 +1,171 @@ ++++ +title = "Building the Ultimate Linux Home Server - Part 2: Docker, Automatic Updates, and File Sharing" +date = 2021-07-25T21:09:00Z +draft = false +summary = "Installing and configuring Docker, Portainer, Watchtower, and Samba on a home server." +tags = ['Guide', 'Self-Hosting', 'Linux'] + +aliases = ['/ultimate-home-server-part-2'] + +[hero] +src = "hero.jpg" +caption = 'Photo by Ian Taylor via Unsplash' ++++ + +> [!important] Disclaimer +> This guide was written in 2021 and reflects my setup and recommendations at the time. Some tools, software versions, and best practices may have changed since then. Consider checking more recent resources alongside this post. + +In [Part 1]({{% ref "/posts/ultimate-home-server-part-1" %}}) we started building a Linux Home Server using a combination of Arch Linux, MergerFS, and SnapRAID. In this part, we are going to be continuing our journey by installing Docker, Portainer, and Watchtower for easy container management, as well as run our first service. + +## What is Docker and why should I use it? + +[Docker](https://www.docker.com/) is an OS-level virtualization project that allows users to install and run applications inside so-called containers, isolated from the operating system and each other. This means that when running a docker container, one does not have to worry about conflicting dependencies, networking, or junk files left behind when removing a service. + +Why should we use Docker instead of another virtualization solution like VMWare, VirtualBox, or KVM? The main reason is performance improvements. While each VM has to simulate its own hardware and run a separate guest operating system, Docker containers instead run as a process in userspace, therefore sharing the host OS' kernel and resources. + +![https://www.docker.com/resources/what-container](docker.png) + +Because of this, containers can boot in seconds, have near bare-metal performance, use way fewer resources than VMs, and are more portable because of their smaller size. + +### Installation + +Installing Docker on Arch Linux is incredibly simple: + +```bash +$ pacman -S docker docker-compose +$ gpasswd -a nick docker +$ systemctl start docker.service +$ systemctl enable docker.service +$ reboot +``` + +After installing it, you can run a test by using the official `hello-world` image. If everything went well, you should see the following output: + +```bash +$ docker run --rm hello-world +Hello from Docker! +This message shows that your installation appears to be working correctly. + +... +``` + +If you want to familiarise yourself with Docker you can experiment by running containers found in the [Docker Hub](https://hub.docker.com/), reading the [docs](https://docs.docker.com/), or taking a look at the [cheat sheet](https://raw.githubusercontent.com/sangam14/dockercheatsheets/master/dockercheatsheet8.png). + +## Portainer + +We could stop here and manage all our applications using the command line, but sometimes using a Web UI is more convenient. This is where [Portainer](https://www.portainer.io/) comes in: It's a web app that makes it easy to manage Docker containers, networks, compose stacks, volumes, and more, all through a user-friendly UI. + +To install it we are going to run the following commands: + +```bash +$ docker volume create portainer_data +$ docker run -d --name=portainer \ + -e PGID=1000 \ + -e PUID=1000 \ + -p 8000:8000 \ + -p 9000:9000 \ + --restart=unless-stopped \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v portainer_data:/data \ + -v /mnt/storage/configs:/configs \ + portainer/portainer-ce +``` + +Here's a brief walkthrough of what each option does: + + - `-d`: Run container in the background (detached) + - `--name=portainer`: The name of the container + - `-e PGID=1000 and -e PUID=1000`: Set environmental variables PGID and PUID to 1000 (your user id, run id to check it) + - `-p 8000:8000`: Expose port 8000 on the host and bind it to port 8000 in the container + - `-p 9000:9000`: Same as above + - `--restart=unless-stopped`: Automatically restart the container if it crashes or when the server boots + - `-v a:b`: Bind directory a on the host to directory b in the container + - `-v /var/run/docker.sock:/var/run/docker.sock`: Give Portainer access to the host's Docker socket + +After running the command, you can access the web UI using a browser at `[local_server_ip]:9000`. In my case, that would be `192.168.1.254:9000`. + +![](portainer.png) + +We're going to come back to Portainer in the next post, where we will be setting up a routing stack to handle our networking. + +## Watchtower + +We can now start and manage already existing containers easily, but what about updating them? One could manually stop, remove, and recreate containers every time they need an update, but that gets tiresome, especially when dealing with a large number of them. That's why we will use [Watchtower](https://github.com/containrrr/watchtower) to automatically update all containers once every week. + +Installation is once again simple: + +```bash +$ docker run -d --name=watchtower \ + -e PGID=1000 \ + -e PUID=1000 \ + --restart=unless-stopped \ + -v /var/run/docker.sock:/var/run/docker.sock \ + containrrr/watchtower \ + --schedule "0 0 4 * * *" \ + --cleanup +``` + +The two new arguments we use are specific to this image, that's why they are placed after its name: + +- `--schedule "0 0 4 * * *"`: Update containers every day at 4 A.M. +- `--cleanup`: Remove old images + +If all goes well, Watchtower should monitor and automatically check for container updates every day. If you want to stop a container from auto-updating, run it with the label `--label=com.centurylinklabs.watchtower.enable=false`. + +## Samba + +Now that we have finished setting up Docker, we can start deploying some actually useful services. The first of these is going to be [Samba](https://github.com/dperson/samba), a tool that allows server volumes to be accessed by other devices on the network. + +Before we begin, we need to create our folder structure and set permissions. + +```bash +$ cd /mnt/storage + +# Create folders and set their owner +$ mkdir configs private public + +# Set default permissions for private and config folders +$ sudo chmod -R 770 configs private +$ sudo setfacl -R -d -m g::rwx configs private +$ sudo setfacl -R -d -m o::- configs private + +# Set default permissions for public folder +$ sudo chmod -R 777 public +$ sudo setfacl -R -d -m g::rwx public +$ sudo setfacl -R -d -m o::rwx public +``` + +Next, open Portainer, go to the *stacks* tab, and add a new stack named `samba`. In the `docker-compose` field, paste the following, and modify it to your liking: + +```yaml +version: '3.9' + +services:stacks* + samba: + image: dperson/samba + container_name: samba + restart: unless-stopped + environment: + - 'TZ=Europe/Athens' + - 'USERID=1000' + - 'GROUPID=1000' + - 'USER1=user1;password1' + - 'USER2=user2;password2' + - 'SHARE1=public;/mount/public;yes;no;yes;all' + - 'SHARE2=user1;/mount/private/user1;yes;no;no;user1' + - 'SHARE3=user2;/mount/private/user2;yes;no;no;user2' + volumes: + - '/mnt/storage:/mount' + ports: + - "139:139/tcp" + - "445:445/tcp" + network_mode: bridge + stdin_open: true + tty: true +``` + +For more information on what each option does, you can check the project's [repository](https://github.com/dperson/samba#configuration). After you are satisfied with your settings, click *deploy the stack*, wait a couple of seconds, and try accessing your new file share from a different computer. + +## Final Thoughts + +By now, you should have installed and configured Docker, Portainer, and Watchtower for container management, along with Samba for all your network storage needs. In the next part, we are going to start exposing our server to the internet using OpenVPN, Cloudflare, and Nginx Proxy Manager. diff --git a/content/posts/ultimate-home-server-part-2/portainer.png b/content/posts/ultimate-home-server-part-2/portainer.png new file mode 100644 index 0000000..76523c1 --- /dev/null +++ b/content/posts/ultimate-home-server-part-2/portainer.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7f0592d0f4ac9a2e723b037ecffba282b42d3adc3d9b118fa45f45b38b16b577 +size 183031 diff --git a/content/posts/ultimate-home-server-part-3/hero.jpg b/content/posts/ultimate-home-server-part-3/hero.jpg new file mode 100644 index 0000000..10403b0 --- /dev/null +++ b/content/posts/ultimate-home-server-part-3/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4eea34440715c9e7583fe734a79c0c5b0b14b5d6d8e9d177b01ca81712925162 +size 882780 diff --git a/content/posts/ultimate-home-server-part-3/index.md b/content/posts/ultimate-home-server-part-3/index.md new file mode 100644 index 0000000..a5c00f5 --- /dev/null +++ b/content/posts/ultimate-home-server-part-3/index.md @@ -0,0 +1,402 @@ ++++ +title = "Building the Ultimate Linux Home Server - Part 3: Cloudflare, OpenVPN, and Nginx Proxy Manager" +date = 2021-08-02T12:17:00Z +draft = false +summary = "Configuring OpenVPN to remotely manage our server, as well as laying the groundwork for other applications using Cloudflare and Nginx Proxy Manager." +tags = ['Guide', 'Self-Hosting', 'Linux'] + +aliases = ['/ultimate-home-server-part-3'] + +[hero] +src = "hero.jpg" +caption = 'Photo by NASA via Unsplash' ++++ + +> [!important] Disclaimer +> This guide was written in 2021 and reflects my setup and recommendations at the time. Some tools, software versions, and best practices may have changed since then. Consider checking more recent resources alongside this post. + +Up until now, we've only been able to access our server while connected to the same network. In this post, we are going to configure OpenVPN so that we can remotely manage it, as well as lay the groundwork for other applications using Cloudflare and Nginx Proxy Manager. + +## Router Configuration + +Before we begin, we are going to need to change a couple of settings on our router. Sadly, not all routers are configured the same way so there is no universal guide. Despite that, most will use the same terminology so it's pretty easy to figure it out. + +### Static IP + +By default, most routers use [DHCP](https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol) which means that they assign dynamic IP addresses to every device that connects to them. However, this makes it difficult to consistently access our server. This is why we are going to give it a static IP. + +First, we need to find our server's MAC address. Open a terminal and run `ifconfig -a`. The network interface we care about should have a line starting with `ether`, `HWaddr`, or `lladdr`. If you have multiple network interfaces (both Ethernet and WiFI for example), you can run `ip route | grep default` to see which one you are using. + +Then, open a browser and go to your router's settings by typing the address of its default gateway (the same address that you get when running `ip route | grep default`). You should have a section named *DHCP*, *Static leases* or something similar. There, you are going to add a new entry, set its IP to whatever you like (in my case `192.168.1.254`) and fill in the above MAC address. + +You might need to restart your server and/or router, but if everything went well, you should now be able to see the line `inet 192.168.1.254` next to the network interface when running `ifconfig -a`. + +### Port Forwarding + +Next, we need to forward some ports to allow a direct connection from the internet to our server. Again, there's no universal guide on how to do this, so you're going to have to do a bit of digging through your router's settings. + +To use OpenVPN, you need to forward port `1194` with the following settings: + +- **Name**: OpenVPN (or anything else you prefer) +- **Internal Port**: 1194 +- **External Port**: 1194 +- **Protocol**: UDP +- **Destination IP**: 192.168.1.254 (or the static IP that you chose) + +I have also forwarded a port for remote SSH connections (both TCP and UDP) in case the VPN goes down, but you should only do this if you have [hardened](https://wiki.archlinux.org/title/security#SSH) your SSH server. + +## Custom Domain + +Now that we have configured our router, it's time to get a custom domain name for our server. You could use a [DDNS](https://en.m.wikipedia.org/wiki/Dynamic_DNS) service like [NoIP](https://www.noip.com/), however, most such websites place several restrictions when using the free plan, making it more worth to just rent a real domain. I personally got mine from [Google Domains](https://domains.google/) for about 12$ a year. + +Next up, we are going to change our nameservers to [Cloudflare](https://dash.cloudflare.com/sign-up)'s in order to take advantage of their caching and security features. This is not mandatory, but it's going to make it easier to fine-tune your domain's settings later on. + +After getting your domain, you should create a Cloudflare account using their free tier and configure your new website. Depending on your registrar, the process of changing your nameservers will be a bit different, so you should probably follow the [official guide](https://support.cloudflare.com/hc/en-us/articles/205195708-Changing-your-domain-nameservers-to-Cloudflare). + +## Cloudflare DDNS + +To create the VPN, we are going to use the docker container [cloudflare-ddns](https://hub.docker.com/r/oznu/cloudflare-ddns/) to make a subdomain always point to our real IP. + +> [!important] Note +> This section is only necessary if your router doesn't support static public IP addresses. + +You first need to create a Cloudflare API key: + +1. Go to https://dash.cloudflare.com/profile/api-tokens +2. Click Create Token +3. Provide the token with a name, for example, `cloudflare-ddns` +4. Grant the following permissions: + - Zone - Zone Settings - Read + - Zone - Zone - Read + - Zone - DNS - Edit +5. Set the zone resources to: + - Include - All zones +6. Complete the wizard and copy the generated token into the `API_KEY` variable for the container. Make sure to note this down somewhere since you won't be able to access it afterwards. + +After getting your API key, go to your Portainer instance, open the *stacks* tab, and add a new stack named `routing`. In the `docker-compose` field paste the following and customize to match your API key and domain: + +```yaml +version: '3.9' + +services: + cloudflare_ddns: + image: oznu/cloudflare-ddns + container_name: cloudflare_ddns + restart: unless-stopped + environment: + - API_KEY=[your-api-key] + - ZONE=example.com + - SUBDOMAIN=ddns + - PROXIED=false + network_mode: bridge +``` + +Click *deploy the stack* and you should notice that a new A record has appeared on the DNS tab of your Cloudflare dashboard. + +I have also added a couple of CNAME records pointing to the original A record specifically for SSH and VPN services: `ssh.example.com` and `vpn.example.com`. + +## OpenVPN + +Now it's finally time to start our VPN. For that, we are going to use the [kylemanna/openvpn](https://hub.docker.com/r/kylemanna/openvpn/) container: + +```bash +# Create a data directory for your configs +$ cd /mnt/storage/configs +$ mkdir openvpn + +# Initialize the configuration files and certificates (change server URL) +$ docker run -v /mnt/storage/configs/openvpn:/etc/openvpn --rm -e PUID=1000 -e PGID=1000 kylemanna/openvpn ovpn_genconfig -u udp://vpn.example.com +$ docker run -v /mnt/storage/configs/openvpn:/etc/openvpn --rm -e PUID=1000 -e PGID=1000 -it kylemanna/openvpn ovpn_initpki + +# Generate a new certificate (change CLIENTNAME) +$ docker run -v /mnt/storage/configs/openvpn:/etc/openvpn --rm -e PUID=1000 -e PGID=1000 -it kylemanna/openvpn easyrsa build-client-full CLIENTNAME + +# Retrieve the certificate (change CLIENTNAME) +$ docker run -v /mnt/storage/configs/openvpn:/etc/openvpn --rm -e PUID=1000 -e PGID=1000 kylemanna/openvpn ovpn_getclient CLIENTNAME > CLIENTNAME.ovpn +``` + +After running these commands, go to Potainer and deploy a new `vpn` stack: + +```yaml +version: '3.9' + +services: + vpn: + image: kylemanna/openvpn + container_name: openvpn + restart: unless-stopped + volumes: + - '/mnt/storage/configs/openvpn:/etc/openvpn' + environment: + - PUID=1000 + - PGID=1000 + ports: + - '1194:1194/udp' + networks: + - vpn + cap_add: + - NET_ADMIN + +networks: + vpn: + name: vpn +``` + +If everything went well, you should be able to download an OpenVPN client from one of these sources and import your certificate. + +- Windows: https://openvpn.net/community-downloads/ +- MacOS: https://openvpn.net/client-connect-vpn-for-mac-os/ +- Android: https://play.google.com/store/apps/details?id=net.openvpn.openvpn +- iOS: https://apps.apple.com/us/app/openvpn-connect/id590379981 +- Linux: use `NetworkManager` and `networkmanager-openvpn`. + +Then, you simply connect to your server from anywhere in the world and access your local network as if you were at home. + +## Nginx Proxy Manager + +By now you should be able to connect and manage your server remotely. But what happens if you want to simply open a browser, go to `subdomain.example.com` and access a self-hosted application? This is where [Nginx Proxy Manager](https://nginxproxymanager.com/) comes in. + +![https://www.cloudflare.com/img/learning/cdn/glossary/reverse-proxy/reverse-proxy-flow.svg](reverse-proxy.png) + +NPM, like any other reverse proxy, allows you to point all of your subdomains to itself and it will automatically manage how each request is routed. For example, both `www.example.com` and `blog.example.com` can point to the proxy's port but get routed to different local servers, one for your website and one for your blog. + +To deploy NPM, we are going to use the container [jlesage/docker-nginx-proxy-manager](https://github.com/jlesage/docker-nginx-proxy-manager). + +```bash +$ cd /mnt/storage/configs +$ mkdir routing +$ mkdir routing/nginx +``` + +Next, we're going to create a custom Nginx config file to change some of the default settings. Make sure to also change the `Local subnets` section to match your own network: + +```nginx {hl_lines=["29-32", "61-64"]} +# run nginx in foreground +daemon off; + +#user root; + +# Set number of worker processes automatically based on number of CPU cores. +worker_processes auto; + +# Enables the use of JIT for regular expressions to speed-up their processing. +pcre_jit on; + +error_log /data/logs/error.log warn; + +# Includes files with directives to load dynamic modules. +include /etc/nginx/modules/*.conf; + +events { + worker_connections 1024; +} + +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + sendfile on; + server_tokens off; + tcp_nopush on; + tcp_nodelay on; + client_body_temp_path /var/tmp/nginx/body 1 2; + keepalive_timeout 90s; + proxy_connect_timeout 90s; + proxy_send_timeout 90s; + proxy_read_timeout 90s; + ssl_prefer_server_ciphers on; + gzip on; + proxy_ignore_client_abort off; + client_max_body_size 0; + server_names_hash_bucket_size 1024; + proxy_http_version 1.1; + proxy_set_header X-Forwarded-Scheme $scheme; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header Accept-Encoding ""; + proxy_cache off; + proxy_cache_path /var/lib/nginx/cache/public levels=1:2 keys_zone=public-cache:30m max_size=192m; + proxy_cache_path /var/lib/nginx/cache/private levels=1:2 keys_zone=private-cache:5m max_size=1024m; + + log_format proxy '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"'; + log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"'; + + access_log /data/logs/default.log proxy; + + # Dynamically generated resolvers file + include /etc/nginx/conf.d/include/resolvers.conf; + + # Default upstream scheme + map $host $forward_scheme { + default http; + } + + # Real IP Determination + + # Local subnets: + set_real_ip_from 10.0.0.0/8; + set_real_ip_from 172.16.0.0/12; # Includes Docker subnet + set_real_ip_from 192.168.1.0/24; + + # NPM generated CDN ip ranges: + include conf.d/include/ip_ranges.conf; + # always put the following 2 lines after ip subnets: + real_ip_header X-Real-IP; + real_ip_recursive on; + + # Custom + include /data/nginx/custom/http_top[.]conf; + + # Files generated by NPM + include /etc/nginx/conf.d/*.conf; + include /data/nginx/default_host/*.conf; + include /data/nginx/proxy_host/*.conf; + include /data/nginx/redirection_host/*.conf; + include /data/nginx/dead_host/*.conf; + include /data/nginx/temp/*.conf; + + # Custom + include /data/nginx/custom/http[.]conf; +} + +stream { + # Files generated by NPM + include /data/nginx/stream/*.conf; + + # Custom + include /data/nginx/custom/stream[.]conf; +} + +# Custom +include /data/nginx/custom/root[.]conf; +``` + +We also need to create a Docker network called `proxy` which will contain all services that will need to be routed to: + +```bash +$ docker network create proxy +``` + +Finally, open the `routing` stack you created earlier in Portainer and edit it like so: + +```yaml +version: '3.9' + +services: + nginx-proxy-manager: + image: jlesage/nginx-proxy-manager + container_name: nginx_proxy_manager + restart: unless-stopped + environment: + - USER_ID=1000 + - GROUP_ID=1000 + - UMASK=002 + - TZ=Europe/Athens + - DISABLE_IPV6=1 + - KEEP_APP_RUNNING=1 + volumes: + - '/mnt/storage/configs/routing/nginx:/config' + - '/mnt/storage/configs/routing/nginx/nginx.conf:/etc/nginx/nginx.conf' + - '/mnt/storage:/static:ro' + ports: + - '80:8080' + - '81:8181' + - '443:4443' + networks: + - proxy + + whoami: + image: containous/whoami + container_name: whoami + restart: unless-stopped + networks: + - proxy + + cloudflare_ddns: + [...] + +networks: + proxy: + external: true + name: proxy +``` + +After deploying the stack, you can open a browser and go to `[your-server-ip]:81` to access NPM's web UI, or `[your-server-ip]:80` to test your installation. However, you still won't be able to actually use the proxy. + +What you first need to do is forward ports 80 and 443 on your router, as well as go to your Cloudflare dashboard and create A, AAAA, or CNAME orange-cloud records for any subdomains you want to use. + +The problem is that some ISPs block HTTP ports, making hosting your apps a bit more difficult. This is why we are going to use a [Cloudflare Argo Tunnel](https://blog.cloudflare.com/tunnel-for-everyone/). + +## Cloudflare Argo Tunnel + +> [!important] Note +> The following section is only necessary if your ISP blocks ports 80 and 443. Otherwise, you can simply forward them and be good to go. + +To create a tunnel, we need to first generate our certificate: + +```bash +$ cd /mnt/storage/configs/routing +$ mkdir cloudflared +$ cd cloudflared +$ docker run cloudflare/cloudflared tunnel login +``` + +Then, follow the instructions on your terminal and save the `cert.pem` file at `/mnt/storage/configs/routing/cloudflared/cert.pem`. + +Finally, edit the `routing` stack again and add the following: + +```yaml +version: '3.9' + +services: + nginx-proxy-manager: + [...] + + whoami: + [...] + + cloudflare_ddns: + [...] + + cloudflare_argo_tunnel: + image: cloudflare/cloudflared + container_name: cloudflare_tunnel + restart: unless-stopped + volumes: + - '/mnt/storage/configs/routing/cloudflared:/etc/cloudflared' + networks: + - proxy + command: tunnel --no-autoupdate --origincert /etc/cloudflared/cert.pem --hostname example.com --no-tls-verify --origin-server-name *.example.com --url https://nginx-proxy-manager:4443 + user: '1000:1000' + +networks: + [...] +``` + +After deploying, you should notice a new AAAA record on your Cloudflare dashboard. If you want to use any other subdomain, you simply need to add a CNAME alias pointing to that record. + +To test if everything is working correctly, add a new CNAME alias for `whoami.example.com` and add a proxy host in NPM with the following settings: + +![](npm-0.png) + +![](npm-1.png) + + +```nginx + proxy_set_header Host $server; + proxy_set_header X-Forwarded-Proto $forward_scheme; + proxy_set_header X-Forwarded-Scheme $forward_scheme; + real_ip_header CF-Connecting-IP; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +``` + +Wait a couple of minutes for the changes to propagate and you should be able to access your new host. + +## Automatic SSL Certificates + +The primary reason why SSL is used is to keep sensitive information sent across the Internet encrypted so that only the intended recipient can access it. One of the main selling points of NPM is automatic SSL certificate management. + +In order to create a wildcard certificate for all of your subdomains, you can follow [this](https://www.reddit.com/r/unRAID/comments/kniuok/howto_add_a_wildcard_certificate_in_nginx_proxy/) guide by [u/Sunsparc](https://www.reddit.com/user/Sunsparc/) on Reddit. After creating your certificate, you should always select it, as well as turn on *Force SSL* and *HTTP/2 Support* on any proxy host you create. + +## Final Thoughts + +After following this guide you should have your own, self-hosted VPN, as well as a reverse proxy, tunneled through Cloudflare. You can now start setting up various other services such as a Plex stack, a Grafana monitoring dashboard, or even your own personal cloud suite using Nextcloud. diff --git a/content/posts/ultimate-home-server-part-3/npm-0.png b/content/posts/ultimate-home-server-part-3/npm-0.png new file mode 100644 index 0000000..506c2cf --- /dev/null +++ b/content/posts/ultimate-home-server-part-3/npm-0.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b69c9534cdb7082237dccd1616d38cab4002341db7541e05434c1d0c261da0a5 +size 29876 diff --git a/content/posts/ultimate-home-server-part-3/npm-1.png b/content/posts/ultimate-home-server-part-3/npm-1.png new file mode 100644 index 0000000..201098e --- /dev/null +++ b/content/posts/ultimate-home-server-part-3/npm-1.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1edf189bb0cc1144197911a0c10a2c0bed379761df9f660f2856d3ce5cd09491 +size 34358 diff --git a/content/posts/ultimate-home-server-part-3/reverse-proxy.png b/content/posts/ultimate-home-server-part-3/reverse-proxy.png new file mode 100644 index 0000000..2f038ec --- /dev/null +++ b/content/posts/ultimate-home-server-part-3/reverse-proxy.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:df6bec79afe8ce830f4f079d841f7509e0b1cfedf374a3d2a0c0f9eb7f9f7663 +size 26228 diff --git a/content/posts/vfio/hero.jpg b/content/posts/vfio/hero.jpg new file mode 100644 index 0000000..b86ab81 --- /dev/null +++ b/content/posts/vfio/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f5253bdc4e2e7d49af46eb9409f8f53d20dd3d917638b0b735c629715676bc77 +size 568836 diff --git a/content/posts/vfio/index.md b/content/posts/vfio/index.md new file mode 100644 index 0000000..26d6b02 --- /dev/null +++ b/content/posts/vfio/index.md @@ -0,0 +1,714 @@ ++++ +title = "Gaming on a VM using a Lenovo Legion 7, Arch Linux, and VFIO Passthrough" +date = 2022-03-30T18:59:00Z +draft = false +summary = "Instructions on how to create a gaming virtual machine on the Legion 7 (2021) using VFIO." +tags = ["Guide", "Virtualization", "Linux", "Windows"] + +aliases = ['/vfio'] + +[hero] +src = "hero.jpg" +caption = 'Photo by Artin Bakhan via Unsplash' ++++ + +Although Linux is not "officially" a gaming platform and is rarely supported by AAA game developers, there are many different ways to play games on it. The most common method is using a compatibility layer such as [Wine](https://www.winehq.org/)/[Proton](https://github.com/ValveSoftware/Proton)/[DXVK](https://github.com/doitsujin/dxvk), however, most anti-cheat solutions don't work on such a layer. + +This is where VFIO passthrough comes in. Given that you have compatible hardware, it is possible to have a VM of any OS with a dedicated GPU and near-native performance. Luckily, Lenovo's Legion 7 (2021) has the perfect hardware setup for such a project. + +In this post, I will give detailed instructions on how to create a Windows gaming virtual machine on the Legion 7 (2021) which hijacks the system's dedicated GPU when it boots and passes it back to the host once it shuts down. + +Note that this setup only works when the laptop is in Hybrid Graphics mode because of its architecture. You could technically adapt it to work while in Discrete Graphics mode as well, but you would have to give up on using your Linux environment while Windows is running. + +## UEFI Settings + +Before doing anything else, you will need to change a couple of settings in the UEFI menu. While the computer is booting, either spam or hold the `F2` key. Once you are in the menu make sure that AMD virtualization is turned on and that the laptop is in Discrete GPU mode. + +You can check if virtualization is working by running `sudo dmesg | grep IOMMU` and `sudo dmesg | grep AMD-Vi`: + +```bash +$ dmesg | grep IOMMU +[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA +[ 0.443759] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported +[ 0.445889] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 +[ 0.454662] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank). +[ 0.470609] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel +$ dmesg | grep AMD-Vi +[ 0.100590] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR0, rdevid:160 +[ 0.100591] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR1, rdevid:160 +[ 0.100592] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR2, rdevid:160 +[ 0.100592] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR3, rdevid:160 +[ 0.443759] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported +[ 0.443794] AMD-Vi: Lazy IO/TLB flushing enabled +[ 0.445889] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 +[ 0.445890] AMD-Vi: Extended features (0x206d73ef22254ade): PPR X2APIC NX GT IA GA PC GA_vAPIC +[ 0.445894] AMD-Vi: Interrupt remapping enabled +[ 0.445894] AMD-Vi: Virtual APIC enabled +[ 0.445894] AMD-Vi: X2APIC enabled +[ 0.470609] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel +``` + +At this point, you will probably be ready to proceed, but if something went wrong you can try adding `amd_iommu=on` to your [kernel parameters](https://wiki.archlinux.org/title/kernel_parameters). + +## Installing QEMU + +After booting your laptop, you're going to have to install the required packages (note that most commands in this post are meant to be used on Arch Linux): + +```bash +$ pacman -Sy nvidia xf86-video-amdgpu samba qemu qemu-arch-extra libvirt virt-manager edk2-ovmf iptables-nft +``` + +If you get a warning about incompatibilities, you should be able to just uninstall the older package. After installation is complete, you will also need to add your user to the `libvirt` group: `sudo usermod -aG libvirt user`. + +Now you can start the necessary services: + +```bash +$ systemctl enable libvirtd.service +$ virsh net-start default +$ virsh net-autostart default +``` + +Finally, restart your laptop and you should be able to run `virt-manager` to manage your virtual machines. + +## Creating a Windows VM + +Before doing any complicated hardware passthrough, it's a good idea to create a normal virtual machine to have as a working reference point. You could use Windows 11, but it's been known to have some performance issues that impact Ryzen CPUs, so I recommend sticking with Windows 10 for now. + +### VirtIO Drivers + +Since I had a dual-boot setup before trying VFIO, I had already installed Windows on a second drive. This meant that I could simply pass through the entire drive to the VM, I just needed to install the [VirtIO](https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/?C=M%3BO%3DD) drivers beforehand. + +### Virt-Manager Configuration + +After installing the VirtIO drivers, boot back into Linux and create a new VM in `virt-manager`: + +1. Select "Manual Install" and make sure the architecture is set to "x86_64" + + ![](virt-manager-1.png) + +2. Set the Operating System to "Microsoft Windows 10". + + ![](virt-manager-2.png) + +3. Allocate as much memory as you need but make sure to leave some for the host as well. You can see how much memory the host is currently using by running `free -m`. Changing the CPU options doesn't really matter since we'll manually change them later. + + ![](virt-manager-3.png) + +4. Do not enable storage for this virtual machine for now. We'll do that during the customization. + + ![](virt-manager-4.png) + +5. Select "Customize configuration before install" and make sure that "Virtual network 'default': NAT" is chosen. + + ![](virt-manager-5.png) + +A new window should pop up. This is where you can customize your virtual machine. Before booting it, you should change a couple of options: + +Under "Overview", Set the Chipset to "Q35" and the Firmware to "UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd". Make sure to click "Apply" before switching pages. + + ![](virt-manager-6.png) + +Under "CPUs", deselect "Copy host CPU configuration" and pick `host-passthrough`. You should also select "Manually set CPU topology" and set 1 Socket, 6 Cores, and 2 Threads (12 total virtual cores), leaving 4 for the host. If you have a different CPU, you should make sure to change these options to fit your configuration. + + ![](virt-manager-7.png) + +Click "Add Hardware > Storage" and add the drive that contains Windows. Set "Bus type" to "VirtIO" and "Cache mode" to "none" for better performance. + +> [!warning] +> Make sure that the drive contains the Windows Bootloader. If it doesn't, you can resize the Windows partition, boot from a rescue USB, and create a Bootloader using BCDBOOT. + + ![](virt-manager-8.png) + +Set the Network Device and Video Device models to "VirtIO". + + ![](virt-manager-9.png) + ![](virt-manager-10.png) + +Finally, click "Begin Installation". You might have to manually add a Boot Entry to the UEFI by pressing the Escape key while the VM is booting, going to "Boot Maintenance Manager > Boot Options > Add Boot Option > Windows Disk > EFI > Microsoft > Boot > bootmgfw.efi". + +If you get a "Boot Device Not Found" Blue Screen, make sure that you have the VirtIO drivers mounted, or try booting with a "SATA" drive instead of "VirtIO". + +After that, if everything is working fine, you can go ahead and shut down the VM. + +## IOMMU Groups + +Next, we need to figure out the IOMMU groups of the NVIDIA GPU. IOMMU refers to the chipset device that maps virtual addresses to physical addresses of your I/O devices (i.e. GPU, disk, etc.). When passing through a device to a VM, you normally need to pass along all other devices in its IOMMU group. + +To check your IOMMU groups, create an `iommu.sh` script with the following content: + +```bash +#!/bin/bash + +for d in /sys/kernel/iommu_groups/*/devices/*; do + n=${d#*/iommu_groups/*}; n=${n%%/*} + printf 'IOMMU Group %s ' "$n" + lspci -nns "${d##*/}" +done +``` + +When you run it, you should see output similar to this: + +```bash +$ ./iommu.sh +IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632] +IOMMU Group 10 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a] +IOMMU Group 11 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) +IOMMU Group 12 04:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a) +IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633] +IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634] +IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632] +IOMMU Group 4 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634] +IOMMU Group 5 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634] +IOMMU Group 6 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632] +IOMMU Group 6 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635] +IOMMU Group 6 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635] +IOMMU Group 6 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a] (rev c5) +IOMMU Group 6 05:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df] +IOMMU Group 6 05:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639] +IOMMU Group 6 05:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639] +IOMMU Group 6 05:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3] +IOMMU Group 6 06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81) +IOMMU Group 6 06:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81) +IOMMU Group 7 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51) +IOMMU Group 7 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51) +IOMMU Group 8 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a] +IOMMU Group 8 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b] +IOMMU Group 8 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c] +IOMMU Group 8 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d] +IOMMU Group 8 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e] +IOMMU Group 8 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f] +IOMMU Group 8 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670] +IOMMU Group 8 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671] +IOMMU Group 9 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104M [GeForce RTX 3070 Mobile / Max-Q] [10de:24dd] (rev a1) +IOMMU Group 9 01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1) +``` +What we mainly care about are the GPU groups, specifically: + +```bash +... +IOMMU Group 9 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104M [GeForce RTX 3070 Mobile / Max-Q] [10de:24dd] (rev a1) +IOMMU Group 9 01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1) +... +``` + +If you only want to pass through the GPU, you can go ahead and skip the following section. However, if you want to pass through more devices that are in less than ideal IOMMU groups, ACS patching can help. + +## Patching the Kernel + +> [!warning] +> Before proceeding, make sure you understand the [risks](https://www.reddit.com/r/VFIO/comments/bvif8d/comment/eppfcf1/) of [ACS patching](https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html)! + +By using the ACS override patch, we can basically force the kernel to falsely expose isolation capabilities for our components and add them to separate IOMMU groups. Luckily for us, we don't need to apply the patch ourselves since there is already an [AUR](https://wiki.archlinux.org/title/Arch_User_Repository) package with the patch pre-applied: [linux-vfio](https://aur.archlinux.org/packages/linux-vfio/). + +Before building the package, make sure to edit your `makepkg` settings so that you use all your CPU's cores. You can do that by editing `/etc/makepkg.conf` and setting the `MAKEFLAGS` line (under "Architecture, Compile Flags") to `-j$(nproc)`. + +```ini +... +#-- Make Flags: change this for DistCC/SMP systems +MAKEFLAGS="-j$(nproc)" +... +``` + +Afterward, install the needed packages using your favorite AUR helper. In my case, the command is: + +```bash +yay -S linux-vfio linux-vfio-headers nvidia-dkms xf86-video-amdgpu +``` + +> [!tip] +> If you get an error about invalid keys, try running `gpg --keyserver hkps://keys.openpgp.org --receive-keys [key]` + +You can now go ahead and make a cup of coffee or grab a snack as the building process takes about 20 minutes. + +You should also be aware that the `amdgpu` module is not automatically loaded in the `linux-vfio` kernel, leading you to a blank screen. This can be easily fixed by creating a `/etc/modules-load.d/display.conf` file: + +``` +amdgpu +nvidia +``` + +Finally, make sure to add `pcie_acs_override=downstream,multifunction` to your kernel's command line parameters. If you are using GRUB, this can be done by editing `/etc/default/grub` and then running `sudo grub-mkconfig -o /boot/grub/grub.cfg`. + +```ini {hl_lines=[8]} +# GRUB boot loader configuration + +GRUB_DEFAULT="" +GRUB_TIMEOUT="1" +GRUB_HIDDEN_TIMEOUT="1" +GRUB_DISABLE_OS_PROBER="false" +GRUB_DISTRIBUTOR="Arch" +GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream,multifunction" +#GRUB_SAVEDEFAULT="false" +#GRUB_DEFAULT="saved" +... +``` + +After rebooting, you can check which kernel is running using the command `uname -a`. If everything went well, you should see that you are using the` linux-vfio` kernel and that most devices are in different IOMMU groups: + +```bash +$ uname -a +Linux eirene 5.14.10-arch1-1-vfio #1 SMP PREEMPT Sat, 06 Nov 2021 18:49:38 +0000 x86_64 GNU/Linux +$ ./iommu.sh +IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632] +IOMMU Group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a] +IOMMU Group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b] +IOMMU Group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c] +IOMMU Group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d] +IOMMU Group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e] +IOMMU Group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f] +IOMMU Group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670] +IOMMU Group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671] +IOMMU Group 11 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104M [GeForce RTX 3070 Mobile / Max-Q] [10de:24dd] (rev a1) +IOMMU Group 12 01:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1) +IOMMU Group 13 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a] +IOMMU Group 14 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) +IOMMU Group 15 04:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a) +IOMMU Group 16 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a] (rev c5) +IOMMU Group 17 05:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df] +IOMMU Group 18 05:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639] +IOMMU Group 19 05:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639] +IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633] +IOMMU Group 20 05:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3] +IOMMU Group 21 06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81) +IOMMU Group 22 06:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81) +IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634] +IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632] +IOMMU Group 4 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634] +IOMMU Group 5 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634] +IOMMU Group 6 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632] +IOMMU Group 7 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635] +IOMMU Group 8 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635] +IOMMU Group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51) +IOMMU Group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51) +``` + +## Creating the Hook Scripts + + +Before doing any passthrough, we need to create the scripts that will allocate the necessary resources to the VM before it boots and de-allocate them after it shuts down. To do that, we are going to be using [libvirt hooks](https://libvirt.org/hooks.html) and The Passthrough Post's [hook helper](https://passthroughpo.st/simple-per-vm-libvirt-hooks-with-the-vfio-tools-hook-helper/). You can find all the needed scripts in this project's [git repository](https://git.karaolidis.com/karaolidis/legion-7-vfio). + +> [!warning] +> If you already have hooks set up, the next step will overwrite them. + +Go ahead and run the following commands: + +```bash +mkdir -p /etc/libvirt/hooks +wget 'https://raw.githubusercontent.com/PassthroughPOST/VFIO-Tools/master/libvirt_hooks/qemu' -O /etc/libvirt/hooks/qemu +chmod +x /etc/libvirt/hooks/qemu +systemctl restart libvirtd.service +``` + +Next, you need to set up the directory structure, like so: + +```bash +$ tree /etc/libvirt/hooks/ +/etc/libvirt/hooks/ +├── qemu +└── qemu.d + └── win10 + ├── prepare + │ └── begin + ├── release + │ └── end + └── started + └── begin +``` + +Scripts in the `prepare/begin` directory will be executed before the VM starts, scripts in the `started/begin` directory will be executed once the VM starts, and scripts in the `release/end` directory will be executed once the VM shuts down. + +### Environment Variables + +The first file we are going to create will contain all of our environment variables, specifically the addresses of the GPU we are going to pass through. Create a `kvm.conf` file in `/etc/libvirt/hooks`: + +```ini +## Virsh devices +VIRSH_GPU_VIDEO=pci_0000_01_00_0 +VIRSH_GPU_AUDIO=pci_0000_01_00_1 +MEMORY=20480 +WIN=/dev/nvme1n1p3 +``` + +The addresses should be the same if you are using a Legion 7 but you can double-check by re-running `iommu.sh`. + +Set `WIN` to the Windows partition if you want to unmount it automatically once the VM starts. + +### Start Script + +Now we are going to create the script that prepares the host for passthrough. Create a `start.sh` file in `/etc/libvirt/hooks/qemu.d/Win10/prepare/begin` with the following contents: + +```bash {hl_lines=[12, "40-43"]} +#!/bin/bash + +set -x + +# Load Variables +source "/etc/libvirt/hooks/kvm.conf" + +# Unmount the Windows drive +umount $WIN || /bin/true + +# Stop LightDM +systemctl stop lightdm.service +sleep 2 + +# Calculate number of hugepages to allocate from memory (in MB) +HUGEPAGES="$(($MEMORY/$(($(grep Hugepagesize /proc/meminfo | awk '{print $2}')/1024))))" + +echo "Allocating hugepages..." +echo $HUGEPAGES > /proc/sys/vm/nr_hugepages +ALLOC_PAGES=$(cat /proc/sys/vm/nr_hugepages) + +TRIES=0 +while (( $ALLOC_PAGES != $HUGEPAGES && $TRIES < 1000 )) +do + echo 1 > /proc/sys/vm/compact_memory ## defrag ram + echo $HUGEPAGES > /proc/sys/vm/nr_hugepages + ALLOC_PAGES=$(cat /proc/sys/vm/nr_hugepages) + echo "Succesfully allocated $ALLOC_PAGES / $HUGEPAGES" + let TRIES+=1 +done + +if [ "$ALLOC_PAGES" -ne "$HUGEPAGES" ] +then + echo "Not able to allocate all hugepages. Reverting..." + echo 0 > /proc/sys/vm/nr_hugepages + exit 1 +fi + +# Unload all Nvidia drivers +modprobe -r nvidia_drm +modprobe -r nvidia_modeset +modprobe -r nvidia_uvm +modprobe -r nvidia + +# Unbind the GPU from display driver +virsh nodedev-detach $VIRSH_GPU_VIDEO +virsh nodedev-detach $VIRSH_GPU_AUDIO + +# Load VFIO kernel modules +modprobe vfio +modprobe vfio_pci +modprobe vfio_iommu_type1 + +# Create looking glass shm +systemd-tmpfiles --create /etc/tmpfiles.d/10-looking-glass.conf + +# Enable CPU governor performance mode +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +for file in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo "performance" > $file; done +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor + +# Isolate host +systemctl set-property --runtime -- user.slice AllowedCPUs=12,13,14,15 +systemctl set-property --runtime -- system.slice AllowedCPUs=12,13,14,15 +systemctl set-property --runtime -- init.scope AllowedCPUs=12,13,14,15 +``` + +Make sure to change the highlighted lines to match your system: + +- Line 12 should stop whatever display manager you are using. +- Lines 40-43 should unload all available Nvidia drivers and their dependencies. You can see them by running `lsmod | grep -i nvidia`. + +### Started Script + +Next, we are going to create the script that re-launches our display manager using only integrated graphics. Create a `lightdm.sh` file in `/etc/libvirt/hooks/qemu.d/win10/started/begin`: + +```bash +#!/bin/bash + +# Uncomment autologin lines in /etc/lightdm/lightdm.conf +sed -i 's/^#autologin-user=/autologin-user=/' /etc/lightdm/lightdm.conf +sed -i 's/^#autologin-user-timeout=/autologin-user-timeout=/' /etc/lightdm/lightdm.conf +sed -i 's/^#autologin-session=/autologin-session=/' /etc/lightdm/lightdm.conf + +# Restart lightdm +systemctl restart lightdm.service + +# Sleep 10 seconds +sleep 10 + +# Comment out autologin lines in /etc/lightdm/lightdm.conf +sed -i 's/^autologin-user=/#autologin-user=/' /etc/lightdm/lightdm.conf +sed -i 's/^autologin-user-timeout=/#autologin-user-timeout=/' /etc/lightdm/lightdm.conf +sed -i 's/^autologin-session=/#autologin-session=/' /etc/lightdm/lightdm.conf +``` + +### Revert Script + +Finally, we are going to create the script that reverts the host once the VM shuts down. This is basically going to be the inverse of the previous script with a couple of small differences. Create a `revert.sh` file in `/etc/libvirt/hooks/qemu.d/win10/release/end`: + +```bash {hl_lines=[31, "34-37"]} +#!/bin/bash + +set -x + +# Load Variables +source "/etc/libvirt/hooks/kvm.conf" + +# Deisolate host +systemctl set-property --runtime -- user.slice AllowedCPUs=0-15 +systemctl set-property --runtime -- system.slice AllowedCPUs=0-15 +systemctl set-property --runtime -- init.scope AllowedCPUs=0-15 + +# Enable CPU governor schedutil mode +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +for file in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo "schedutil" > $file; done +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor + +# Delete looking glass shm +rm /dev/shm/looking-glass + +# Unload VFIO kernel modules +modprobe -r vfio_pci +modprobe -r vfio_iommu_type1 +modprobe -r vfio + +# Rebind the GPU to display driver +virsh nodedev-reattach $VIRSH_GPU_VIDEO +virsh nodedev-reattach $VIRSH_GPU_AUDIO + +# Read our nvidia configuration before starting our graphics +nvidia-xconfig --query-gpu-info > /dev/null 2>&1 + +# Load all Nvidia drivers +modprobe nvidia_drm +modprobe nvidia_modeset +modprobe nvidia_uvm +modprobe nvidia + +# Dealloc hugepages +echo 0 > /proc/sys/vm/nr_hugepages + +# Mount the Windows drive +mount -a || /bin/true +``` + +- Line 31 wakes up the GPU by querying its config, it might be redundant. +- Lines 34-37 load all drivers that were unloaded in the start script. + +You should also make sure that the scripts are owned by `root` and have execute permissions before moving on to the next section. + +## Passing Through Devices + +Now is finally time to pass through the GPU. Open `virt-manager`, select your VM, and click the "Add Hardware > PCI Host Device" button. You should see a large list containing the same devices as shown when running the `iommu.sh` script. + +### Nvidia GPU + +Select your GPU, as well as any other devices that were in the same IOMMU group, and add them to the VM. + +![](virt-manager-11.png) + +You can also add any other devices you like such as network cards or a USB controller, but make sure to also add their IOMMU neighbors. + +### Virtual Battery + +When using mobile graphics cards, the Nvidia driver wants to check the status of the power supply. Since we are using a VM, no battery is present, and the driver shows the infamous "Error 43". + +To fix this, download a [custom ACPI table](https://git.karaolidis.com/karaolidis/legion-7-vfio/raw/branch/master/resources/SSDT1.dat) and edit your VM's XML like so: + +```xml + + ... + + + + + +``` + +## Tuning Performance + +You could technically now boot the VM, install Looking Glass, and be good to go. However, there are a couple of things you can do to greatly improve performance. + +### CPU Pinning + +CPU Pinning is the assignment of a process or task to a specific CPU core. This has the advantage of significantly increasing cache utilization, and therefore performance. + +To see which cores you need to pin, you can use the `lstopo` utility: + +![](lstopo.png) + +Next up, edit your VM XML, and add the following parameters (make sure to pin neighboring cores): + +```xml +... + 12 + + + + + + + + + + + + + + + ... + + + + +... +``` + +If you are using an AMD CPU, you will also want to enable SMT: + +```xml +... + + ... + + +... +``` + +### Hugepages + +After pinning your CPU cores, you will also want to enable huge pages to reduce memory latency. The hook scripts should handle memory allocation automatically, but you still need to edit your XML: + +```xml +... + 20971520 + 20971520 + + + +... +``` + +### Hyper-V Enlightenments + +You should also enable some Hyper-V enlightenments to help the guest OS handle nested virtualization. You can find more information on what each option does [here](https://libvirt.org/formatdomain.html#elementsFeatures). + +```xml +... + + + + + + + + + + + + + + + + + + + +... +``` + +### Disabling Memballoon + +The VirtIO memballoon device allows the host to reclaim memory from a running VM. However, this functionality comes at a performance cost, so you can disable it by editing the `` tag in your XML like so: + +```xml +... + +... +``` + +## Setting up Looking Glass + +If you were to boot the VM now, you would notice that even though the GPU is passed through correctly, it is unusable. This happens because the Nvidia drivers expect a display to be connected, but we don't actually have one. + +This is where a [Dummy HDMI plug](https://www.amazon.co.uk/gp/product/B07YMTKJCR/) can come in handy. This little device can be used to trick the GPU into thinking a display is connected. The resolution of the plug doesn't really matter since we can set a custom one from the Nvidia control panel anyway. + +We can then use [Looking Glass](https://looking-glass.io/) to hijack the display signal and pass it back to the host with minimal latency. Download the [host application](https://looking-glass.io/downloads) and install it on your Windows VM. Once you're done, shut the VM down and follow the instructions below to finish the configuration. + +### Video Streaming + +First, you need to create the shared memory config. Create a new `/etc/tmpfiles.d/10-looking-glass.conf` file with the following contents: + +``` +f /dev/shm/looking-glass 0666 root libvirt - +``` + +You should also make sure that your user is in the `libvirt` group. After that, edit your XML and add the following in the `` section: + +```xml +... + + + 64 +
+ +... +``` + +Once that's done, you should be able to remove any other graphics devices from your VM, plug the Dummy HDMI, and open the looking glass client on your host. If everything was set up correctly, you should see the Windows lock screen. + +If you also want to pass through mouse and keyboard input using Looking Glass, you can simply add a new "Display Spice" device and set its model type to "none". You should also remove any "Tablet" devices you might have. + +Finally, to enable clipboard sharing, edit your `spicevmc` channel like so: + +```xml +... + + +
+ +... +``` + +## Streaming Audio + +Now that video passthrough is configured, there's only one step left: audio passthrough. First, edit your QEMU configuration at `/etc/libvirt/qemu.conf` and add your user id. You can find it using the `id` command. + +```xml +... +user = 1000 +... +``` + +Reboot to apply the changes. Next, edit your XML once again, and add the following in the devices section: + +```xml +... + + + + +... +``` + +If everything went well, you should now be able to hear your Windows VM's audio outputs on your host. + +## Results + +It is not often that you find a laptop with hardware capable of VFIO passthrough, so seeing this one work so flawlessly was a treat: + +![](windows.png) + +![](linux.png) + +After some not-so-thorough benchmarking, I can say that I get around 75% of the performance compared to bare-metal Windows. This is not that big of a problem for normal usage, however, if I ever need that extra boost, I can simply boot Windows from GRUB and be good to go. + +Was there a point in wasting all this time on such a fiddly setup? Maybe. Was it fun? Definitely. + +## Resources + +A very special thanks to the authors and contributors of the following guides and blog posts: + +- [Arch Wiki: PCI passthrough via OVMF](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF) +- [Maagu Karuri's VFIO Guide](https://gitlab.com/Karuri/vfio) +- [Bryan Steiner's GPU Passthrough Guide](https://github.com/bryansteiner/gpu-passthrough-tutorial) +- [Mathias Huber's PCI Passthrough](https://mathiashueber.com/pci-passthrough-ubuntu-2004-virtual-machine/) and [Optimization Guides](https://mathiashueber.com/performance-tweaks-gaming-on-virtual-machines/) +- [Shariar Shovon's VirtIO Drivers Guide](https://linuxhint.com/install_virtio_drivers_kvm_qemu_windows_vm/) +- [Sebastiaan Meijer's Libvirt Hook Helper](https://passthroughpo.st/simple-per-vm-libvirt-hooks-with-the-vfio-tools-hook-helper/) +- [The Looking Glass Team](https://looking-glass.io/docs/B5.0.1/install/#spice-server) diff --git a/content/posts/vfio/linux.png b/content/posts/vfio/linux.png new file mode 100644 index 0000000..6f456da --- /dev/null +++ b/content/posts/vfio/linux.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8b22b8d8cdbea0e56e7344fc71b05c1d128193eeac87c0a67febe63c5a8ea77a +size 944761 diff --git a/content/posts/vfio/lstopo.png b/content/posts/vfio/lstopo.png new file mode 100644 index 0000000..2ade839 --- /dev/null +++ b/content/posts/vfio/lstopo.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:86a4ebf7572702ddcd36d34b7ba883dfe4c578ae05d1c6c0faf9c90b319c2175 +size 179667 diff --git a/content/posts/vfio/virt-manager-1.png b/content/posts/vfio/virt-manager-1.png new file mode 100644 index 0000000..8a5b197 --- /dev/null +++ b/content/posts/vfio/virt-manager-1.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dbdd4c5b98705030ec21f978139a161f9d57a3c022f21f1df305d0e11c901b26 +size 27558 diff --git a/content/posts/vfio/virt-manager-10.png b/content/posts/vfio/virt-manager-10.png new file mode 100644 index 0000000..e539274 --- /dev/null +++ b/content/posts/vfio/virt-manager-10.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bb27e8368d74715b477177d782a797f33cf72636e1b82093b9dea5afe1a13ce5 +size 47909 diff --git a/content/posts/vfio/virt-manager-11.png b/content/posts/vfio/virt-manager-11.png new file mode 100644 index 0000000..ba9c9ec --- /dev/null +++ b/content/posts/vfio/virt-manager-11.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e40fe02a4e4eac28d0e3f0712762859c485f99787383a49602d3360e54a72b3c +size 102420 diff --git a/content/posts/vfio/virt-manager-2.png b/content/posts/vfio/virt-manager-2.png new file mode 100644 index 0000000..601a975 --- /dev/null +++ b/content/posts/vfio/virt-manager-2.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:44eab7b8e8d6cca6c86787c6eb23907cc8d6e4bf667b61cec53934bafc918b9e +size 17119 diff --git a/content/posts/vfio/virt-manager-3.png b/content/posts/vfio/virt-manager-3.png new file mode 100644 index 0000000..4f97d0f --- /dev/null +++ b/content/posts/vfio/virt-manager-3.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b5459a3d1305ca8b1f6c702d29fa6c0639aa32310141fc049886b04eacba5bac +size 20502 diff --git a/content/posts/vfio/virt-manager-4.png b/content/posts/vfio/virt-manager-4.png new file mode 100644 index 0000000..544e007 --- /dev/null +++ b/content/posts/vfio/virt-manager-4.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:22b24d22c3e042970ebd8bb594e41c8967d7bf504a31d45e938d74f5d293cbde +size 21541 diff --git a/content/posts/vfio/virt-manager-5.png b/content/posts/vfio/virt-manager-5.png new file mode 100644 index 0000000..21de4c2 --- /dev/null +++ b/content/posts/vfio/virt-manager-5.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6c49da401bad4ed925180ac1faf1b062ad706d2c490689f3dcf5412510fd98a9 +size 27660 diff --git a/content/posts/vfio/virt-manager-6.png b/content/posts/vfio/virt-manager-6.png new file mode 100644 index 0000000..5a74baa --- /dev/null +++ b/content/posts/vfio/virt-manager-6.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1c016478e8875f1419fefc40bdbba8a218c369b968594c3621af97d83055cade +size 64560 diff --git a/content/posts/vfio/virt-manager-7.png b/content/posts/vfio/virt-manager-7.png new file mode 100644 index 0000000..0db3fd7 --- /dev/null +++ b/content/posts/vfio/virt-manager-7.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bf0536a742e911c53675c03ba46c134cd2449c1cc44f23ecb74dbd909bd74bfc +size 61615 diff --git a/content/posts/vfio/virt-manager-8.png b/content/posts/vfio/virt-manager-8.png new file mode 100644 index 0000000..18dde54 --- /dev/null +++ b/content/posts/vfio/virt-manager-8.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9b5851365ea8413a21cee7cee6507e0008a02e22a87c8439c2a22d354de293c1 +size 51726 diff --git a/content/posts/vfio/virt-manager-9.png b/content/posts/vfio/virt-manager-9.png new file mode 100644 index 0000000..6861099 --- /dev/null +++ b/content/posts/vfio/virt-manager-9.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6349f45d0edc2af398596569a7fb6a3a814b1cb4a0f8c902d8c623515efc0967 +size 58217 diff --git a/content/posts/vfio/windows.png b/content/posts/vfio/windows.png new file mode 100644 index 0000000..be6dafa --- /dev/null +++ b/content/posts/vfio/windows.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2d42de91625d6fd988faf44ce0f14a98b6fc47b5d516b439fe9e7d2d45614da6 +size 2356316 diff --git a/content/posts/windows-and-linux-on-ntfs/archinstall.png b/content/posts/windows-and-linux-on-ntfs/archinstall.png new file mode 100644 index 0000000..5cb3e24 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/archinstall.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:74394ad5b7d03190ed274f246994ff5ffb6c20091e1f1c84b313b07a711008b2 +size 8505 diff --git a/content/posts/windows-and-linux-on-ntfs/fdisk.png b/content/posts/windows-and-linux-on-ntfs/fdisk.png new file mode 100644 index 0000000..30fce28 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/fdisk.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:24c51ce02385540c9a6eb3afd4fe4c81b388ff1bb76ca5d17ef2f10a005ebe71 +size 11265 diff --git a/content/posts/windows-and-linux-on-ntfs/hero.jpg b/content/posts/windows-and-linux-on-ntfs/hero.jpg new file mode 100644 index 0000000..88e03b6 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/hero.jpg @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:073ab644b82974ee2b335e9ef0d3cf18039a344e51a98279125035d93c95615e +size 1452662 diff --git a/content/posts/windows-and-linux-on-ntfs/index.md b/content/posts/windows-and-linux-on-ntfs/index.md new file mode 100644 index 0000000..26b2f1d --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/index.md @@ -0,0 +1,96 @@ ++++ +title = "Installing Windows and Linux on the Same Partition" +date = 2022-01-09T15:06:00Z +draft = false +summary = "Have you ever wanted to make God cry? If that's the case then this is the perfect post for you!" +tags = ["Experiment", "Guide", "Linux", "Windows"] + +aliases = ['/windows-and-linux-on-ntfs'] + +[hero] +src = "hero.jpg" +caption = 'Photo by benjamin lehman via Unsplash' ++++ + +Have you ever wanted to make God cry? Perhaps you want to take revenge on one of your Computer Scientist friends? Maybe you just hate your laptop and want to see it suffer? If that's the case then this is the perfect post for you! + +With the help of Paragon's new [NTFS3 driver](https://www.kernel.org/doc/html/latest/filesystems/ntfs3.html), it is now possible to install Linux on an NTFS partition. So, why not take it to the next level and install Windows on that partition as well? + +> [!warning] Disclaimer +> I shouldn't have to say this, but DO NOT do this on a bare-metal system. There are several known issues, such as the system bricking itself after a few reboots. I am only doing this as an experiment. + +## Requirements + +- [Arch Linux Installation ISO](https://archlinux.org/download/) +- [Microsoft Windows Installation ISO](https://www.microsoft.com/software-download/windows10ISO) +- Optional: If you are insane enough to try this on a bare-metal system, you will also need a USB stick with the above ISOs burned onto it +- A lot of patience + +## Steps + +1. Boot using the Arch Install ISO. + +2. Format the target drive. In my case, that's `/dev/sda`. It needs to have a GPT partition table and 2 partitions in total: + - 1GB EFI Filesystem + - The rest of the drive (leave it unallocated for now) + + ![](fdisk.png) + +3. Shut down your system and boot using the Windows ISO. + +4. Make sure to select "Custom: Install Windows only (advanced)". + + ![](windows-installation-type.png) + +5. Select the Unallocated Space we created earlier and finish the installation. + + ![](windows-drive.png) + +6. Once the installation is complete, shut the system down and boot into the Arch ISO once again. + +7. Mount the Windows partition as `/mnt/archinstall`. + +8. Run the `archinstall` installation script. + + ![](archinstall.png) + +9. When asked about which drives to configure, make sure to not select anything. + +10. I also decided to use GRUB as my bootloader since I am more familiar with it, but you can use whatever you like. + + ![](linux-drive.png) + +11. The install script will fail during GRUB's installation, so we must continue manually. + +12. Change root into the installation folder and install GRUB. + + ![](linux-chroot.png) + +13. Add the windows bootloader as a custom GRUB entry by editing `/etc/grub.d/40_custom`. Make sure to replace `{UUID}` with your disk's UUID. You can get it by running `blkid /dev/sda1`. + +``` +menuentry 'Windows 10' { + search --fs-uuid --no-floppy --set=root {UUID} + chainloader (${root})/EFI/Microsoft/Boot/bootmgfw.efi +} +``` + +14. Add `rootfstype=ntfs3` to your kernel parameters. You can do that by editing `/etc/default/grub` and appending it to `GRUB_CMDLINE_LINUX_DEFAULT`. + +15. Run `grub-mkconfig -o /boot/grub/grub.cfg` to regenerate the GRUB configuration. + +16. Shut down your system. If everything went well, you should be able to boot into both Windows and Linux from the same partition. + +17. I recommend backing up your drive at this point since you can boot about 5 times before your system nukes itself. + +18. Done! + +![](linux.png) + +![](windows.png) + +## Conclusion + +If you have successfully followed this post... why? Go see a therapist. + +Original idea by [u/fabi_sh](https://www.reddit.com/user/fabi_sh/) on [Reddit](https://www.reddit.com/r/archlinux/comments/qwsftq/arch_linux_on_ntfs3/). diff --git a/content/posts/windows-and-linux-on-ntfs/linux-chroot.png b/content/posts/windows-and-linux-on-ntfs/linux-chroot.png new file mode 100644 index 0000000..9e97dbe --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/linux-chroot.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fce84a50fa623299857e21b6a5509b6ad05d9c5b868ca9e6ee23858a3c7e0c43 +size 14440 diff --git a/content/posts/windows-and-linux-on-ntfs/linux-drive.png b/content/posts/windows-and-linux-on-ntfs/linux-drive.png new file mode 100644 index 0000000..6ce8d20 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/linux-drive.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:35a9a1fd580608cd8d8e1ef0b89b691960a0301e66e6ed2876e10719a790772e +size 4470 diff --git a/content/posts/windows-and-linux-on-ntfs/linux.png b/content/posts/windows-and-linux-on-ntfs/linux.png new file mode 100644 index 0000000..71cec78 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/linux.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:216e6edec25fb476eca30c5f39cd510f363b17a76a9df9c32d908749f62f9724 +size 14141 diff --git a/content/posts/windows-and-linux-on-ntfs/windows-drive.png b/content/posts/windows-and-linux-on-ntfs/windows-drive.png new file mode 100644 index 0000000..b820a46 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/windows-drive.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bede465e62de817f73bb35b3db0977d1e3f220f43f0c878cae01622038500ccf +size 31075 diff --git a/content/posts/windows-and-linux-on-ntfs/windows-installation-type.png b/content/posts/windows-and-linux-on-ntfs/windows-installation-type.png new file mode 100644 index 0000000..16eb814 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/windows-installation-type.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:76ac056b6b7c7bbc37b0edd0e284fdec269c65a012b1016a546d6e2e2da8b532 +size 23991 diff --git a/content/posts/windows-and-linux-on-ntfs/windows.png b/content/posts/windows-and-linux-on-ntfs/windows.png new file mode 100644 index 0000000..2e31cd1 --- /dev/null +++ b/content/posts/windows-and-linux-on-ntfs/windows.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:95743388e5defb2fc361bf664434b935476d93f4884008f8f2aa7fd289a5cddb +size 55792 diff --git a/hugo.toml b/hugo.toml index 6c78cef..ace1cb1 100644 --- a/hugo.toml +++ b/hugo.toml @@ -2,6 +2,7 @@ baseURL = 'https://www.karaolidis.com/' languageCode = 'en-us' title = 'Jupiter Lab' theme = 'caldwell' +enableEmoji = true [[params.navigation]] name = "Blog" @@ -22,3 +23,14 @@ url = "mailto:nick@karaolidis.com" [[params.footer]] name = "Socials" url = "https://social.karaolidis.com" + +[[params.footer]] +name = "Source Code" +url = "https://git.karaolidis.com/karaolidis/blog" + +[taxonomies] + tag = 'tags' + +[markup] +[markup.highlight] + noClasses = false diff --git a/themes/caldwell b/themes/caldwell index d9dcea0..9905066 160000 --- a/themes/caldwell +++ b/themes/caldwell @@ -1 +1 @@ -Subproject commit d9dcea0e41e08c7fce550fb8caf1979c729263f8 +Subproject commit 990506604684283bd4a5acee925c4d5392a043b7