Geek

French version is available on LinuxFr. Comments can be posted on LinkedIn. Introduction This post is a follow-up to my previous articles: Self-hosting your AIs: Generalities Self-hosting your AIs: Hardware and Inference Optimization Self-hosting your LLMs (AIs): CPU+RAM overflow and Mixture-of-Experts We’ve covered the theory, the hardware, and you’ve probably emptied your savings accounts and built your intelligent electric space heater by now. You now have an LLM tuned to perfection. Now we come to the most enjoyable part: actually using the models. In other words, we are going to put user interfaces (frontends) in front of our inference engine to actually put them to work. ...

French version is available on LinuxFr. Comments can be posted on LinkedIn. Introduction This article is a follow-up to my previous entries: Self-hosting your AIs: General Principles Self-hosting your AIs: Hardware and Inference Optimization In this entry, we’re going to talk about running LLMs with little to no VRAM. In other words, we’re going to try and shove a large round peg into a small square hole. Like me, you might have noticed something strange: there are plenty of YouTube videos, LinkedIn posts, etc., telling you that running LLMs on a museum piece (like an Nvidia GTX 1060 6GB with DDR4) is super simple. They explain that it’ll run lightning-fast, work perfectly, and so on. Apparently, you just need to find the right “magic parameters,” and poof! You’ll never need Claude Fable 5 again! 🎉 ...

French version available on LinuxFr.org. Comments can be posted on LinkedIn. Introduction Comrades, digital proletarians, penguins—unite! Against the rent-seeking of proprietary AI, let us arm our GPUs! Let’s seize the means of intelligence production! Alright, soapbox aside. Let’s talk about LLM and hardware. This article is essentially a deep dive into my previous article on self-hosting AI. To get the most out of this, you should be familiar with: Linux in general Docker The PCIe bus Some basic concepts regarding self-hosted LLMs (otherwise, this is going to sting like hell). In my previous article, I suggested a few hardware solutions. Since then, I’ve gathered more data, played around much more with my toys, and—as with almost everything in the current AI world—things have moved surprisingly fast. I’ll also dive deeper into the two hardware setups I’ve tested. Each has its own pitfalls that are well worth mentioning. ...

French version available on LinuxFr.org. Comments can be posted on LinkedIn. Introduction If you believe LinkedIn gurus, AI is the new technological revolution, the new gold rush, and so on. Soon, software engineers will no longer be needed! And if you believe my YouTube recommendations, AI is the latest trendy scam, an unprecedented economic bubble inflating, and an ecological disaster ¯\(ツ)/¯. In any case, in IT professions, the subject has become difficult to avoid. ...

Context Summer is coming. Summer is hot. Gaming on desktop computer makes my office hotter. My laptop on the other hand doesn’t make it that much hotter. My basement is cold. Therefore my Proxmox server is cold, and it doesn’t make my office hotter. Idea Put my video card in my Proxmox server and play remotely using Steam Remote Play. Putting everything in a LXC container allows to keep things nice, tidy and separate from the host system as much as possible. The main goal for me is not security (you have to give the container access to a lot of things), but simply handling backups in a generic manner. It also allow me to snapshot the container easily before trying some dumb idea on it. ...

Introduction I currently have to teach someone how to use the Linux shell (bash). I have to do it remotely. The obvious first step is to create a sandbox virtual machine running Linux and create accounts for them on it. I have a backup of the VM so they can freely make a mess if they want to. But to guide them, I need to have a way to see what they are inputting and what they did input. I could use videoconferencing with screen sharing. But I like letting them do most of the work, so they spend a lot of time looking for info and thinking hard by themselves. So videoconferencing with them all the time would be a waste of my time. Also videoconferencing doesn’t allow me to see what they previously inputted. ...

Introduction I’m a programmer by trade and I’ve been helping my manager recruiting for quite some times. By helping recruiting, I mean separating good applicants from bad ones by having a technical interview with them. Resumes are often full of approximations and flat-out lies. The idea here is to figure out what their actual programming skills are. And I needed a tool to do that efficiently and remotely. ...

On Debian 10 (buster) and Ubuntu 20.04, switching to zsh and liquidprompt, the quick and easy way: 1 2 3 4 5 sudo apt install zsh liquidprompt chsh -s /usr/bin/zsh # restart your session liquidprompt_activate # close your shell and re-open it

Here is the recipe I’ve made recently to integrate Kivy 2.0.0 in Yocto (dunfell). It’s probably far from perfect, but I guess it can help other Yocto+Kivy users like me. Note that I had to patch the setup.py of Kivy to work around a bug. This bug has been fixed now, so this patch won’t be useful for future versions. SUMMARY = "Recipe to embedded the Python PiP Package Kivy" HOMEPAGE = "https://kivy.org/" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://LICENSE;md5=f0c851d60495c7e27225a819e179208a" PYPI_PACKAGE = "Kivy" USE_WAYLAND = "1" export USE_WAYLAND inherit dos2unix pypi setuptools3 SRC_URI += " \ file://0001-setup.py-Fix-get_modulename_from_file.patch \ " SRC_URI[md5sum] = "410d3968c7f45171285bcac76bbad265" SRC_URI[sha256sum] = "d25e44eb44e43762b2fd0c5874e51954e0f1181fd9800d8a6756be6d084812d8" DEPENDS += " \ gstreamer1.0 \ gstreamer1.0-python \ libsdl2 \ libsdl2-image \ libsdl2-mixer \ libsdl2-ttf \ pango \ python3-cython-native \ " RDEPENDS_${PN} = " \ gstreamer1.0 \ gstreamer1.0-python \ libsdl2 \ libsdl2-image \ libsdl2-mixer \ libsdl2-ttf \ pango \ python3-docutils \ python3-fcntl \ python3-image \ python3-kivy-garden \ python3-pillow \ python3-pygments \ " do_install_append() { # drop the examples # XXX(Jflesch): by default they shouldn't have been built by # Kivy's setup.py. Therefore they shouldn't have been installed # either. Yet somehow here they are. rm -rf ${D}/usr/share } Kivy-garden is also required by Kivy, so I also had to make a recipe for it. You can download the complete recipes and patches for Kivy and Kivy-garden here: yocto_recipe_kivy.tar.gz. ...

Use Ubuntu 20.04.1 (Raspbian is only available as armhf images ; we need arm64). In /etc/modules, add libceph and rbd Install K3s Clear the disks you want to use as Ceph storages As root: 1 2 3 sudo apt install ceph-common git clone https://github.com/rook/rook.git cd rook Switch to the raspbernetes images instead of the default ones. Currently the default ones are not all built as multi-arch yet and therefore don’t all work on arm64. ...