Author: dbrand666

Posted on

Setting up ZFS RAIDZ(N) with missing drives

I’m planning to migrate from an old RAIDZ1 pool of three drives to a new RAIDZ2 pool with five. I was thinking that I’d start by setting up four drives in a degraded configuration and add the fifth when I get around to buying it. I figured I could create a loopback drive (or the FreeBSD equivalent, set up the pool, and remove that drive before adding any actual data.

The problem is that I’m using TrueNAS and it, probably wisely, doesn’t let me select that loopback drive when it’s time to set up the pool. I did quite a bit of searching and everyone says that there’s definitely no way to do it via the UI.

Since I’m not afraid of the command line, I tried looking around to find out exactly what parameters TrueNAS uses when it creates the pool. Even though I’m cutting corners, I still wanted to stay as close as possible to a standard configuration. I found these old instructions but I wasn’t convinced the zpool create command would give me exactly the same options that TrueNAS would use, especially since I wanted things like compression and encryption like my old drive had.

So I decided to take another approach.

I knew that it’s possible to expand a RAIDZ array by replacing drives with larger drives. The available space stays unchanged as drives are upgraded but once the last drive is upgraded the array takes on the size of the new drives.

I happened to have a really old drive laying around unused. Really old. 640GB! But it didn’t really much matter. I wasn’t going to be trusting it with any data. I installed that drive, went in to create my RAIDZ2 pool, and selected it along with my 4 “real” drives. All good. I now had a RAIDZ2 pool that was ready to hold about 1.5TB of data (3x640GB).

Next step was to replace that drive with a large loopback drive:

# Create a sparse file to hold the loopback device
truncate -s <disk size, eg 8t> disk.img
# Create the loopback device
mdconfig disk.img
(it output a device name, eg. md0)
zpool replace <pool name> <little drive> <loopback drive>

The pool now showed the full capacity expected of the larger drives. Then I removed that loopback drive from the pool, deleted the loopback device, and the backing file:

zpool detach <pool name> <loopback drive, eg. md0>
mdconfig -du <loopback drive, eg. md0>
rm disk.img

The capacity was unchanged (still the right size) but the pool state showed DEGRADED, as expected.

When the new drive arrives, I’ll add it. Probably best to do it from the TrueNAS GUI. From “Dashboard” scroll down to the pool and select the pool status gear icon. Click on the offline drive and choose Replace.

Posted on

Controlling an Onkyo receiver with a Pi

With my media player/voice assistant running on a Pi Zero, I noticed that my stereo cabinet was a bit warm. The Pi doesn’t use much power but my receiver does. I used to control power to the receiver with an X10 appliance module but I’m phasing that out.

I found a few pages describing how to control Onkyo equipment using their Onkyo RI “Remote Interactive” protocol. I built the interface cable and use the Python code from here and codes from here. I also had to install pigpiod. Use sudo systemctl enable pigpiod and sudo systemctl start pigpiod to start the service (for now) and enable it (after reboots.

Test it from the command line to ensure that the receiver turns on/off as expected.

Next step was to integrate with squeezelite.

I found this man page which documents a -C option which will close the output device after a specified timeout if idle. I set it to half an hour. This works together with the -S option which runs a script when the output device turns on/off. Unfortunately, at this time at least, the version of squeezelite in the repo is too old. My quick fix was to replace the binary /usr/bin/squeezelite with the latest from here which was squeezelite-2.0.0.1486-armhf.tar.gz.

Here’s the script I’m passing to squeezelite’s -S option (my receiver is a TX-8020):

#!/bin/bash
cd .../onkyo-rpi || exit 1
if [ $1 = 0 ]; then
        python main.py --gpio 26 0x420
else
        python main.py --gpio 26 0x02f
fi

I caller the script onkyo-power.sh. Test that it turns the receiver on when called with a 1 argument and off when called with a 0.

A few other notes…

I first assumed that a native C code implementation would be more reliable. I used onkyoricli which I found here. I had no luck with it and after hooking up an oscilloscope to check the output, I saw that it was distorted. The code is correct but the waveforms seem to be getting interrupted by the Linux scheduler and end up with gaps in them. The python package doesn’t try to generate the waveforms itself but instead depends on the pigpiod daemon which seems to handle the realtime issues correctly.

Another issue I ran into is that as soon as I hooked up the RI cable, my audio output was garbled. I tried using a better power supply for the Pi but it didn’t help. I noticed that the noise started as soon as I attached the ground pin on the Pi to the RI cable – attaching only the signal pin was fine. When I checked the signal between the Pi and the RI cable ground I could see a 60Hz signal between them. I believe the audio connection to the receiver is grounded and that I was looking at a ground loop between that connection and the RI port. I ended up leaving off the redundant ground connection between the RI port and the Pi, leaving only the signal pin connected.

Posted on

Home assistant media player with voice input

I’ve been running piCorePlayer for a long time, first with LMS and more recently with Music Assistant. It’s really simple to set up and has many interesting features (like an optional built-in LMS) that I wasn’t using anyway.

I’ve been watching Home Assistant’s Year of Voice project with interest and was about to order an Atom Echo from M5Stack when I realized that the music players I already have scattered around the house should be able to process voice too without much trouble.

First step was to replace piCorePlayer with a Raspberry Pi OS so I’d be working with a more conventional Linux environment. I started with Raspberry Pi Imager to create an SD card. I installed the Lite version since I wasn’t planning on using the UI and configured SSH. Once it booted up I saw that it had already expanded storage so I was ready to start installing the player.

A couple of things I had to get out of the way first. I’m still using that cheap Ethernet adapter so I had to follow this guide first and reboot. Next up, I noticed that the squeezelite package wants to install all kinds of UI things. I thought it would be a good idea to disable all of that before I got started. Create a file named /etc/apt/apt.conf.d/99_norecommend and enter the following settings:

APT::Install-Recommends "false";
APT::AutoRemove::RecommendsImportant "false";
APT::AutoRemove::SuggestsImportant "false";

That will keep apt from installing recommended or suggested packages (like an X11 server).

Next, we can install squeezelite. You can type this all on one line or just cut/paste the whole block:

sudo apt-get update &&
sudo apt-get upgrade &&
sudo apt-get install squeezelite

You may want to go into /etc/default/squeezelite and see if you want to change anything. I changed the player name and increased the ALSA buffer size as recommended here.

Reboot and we should see the player.

The next step is to install Wyoming Satellite. Follow the instructions on that page. I’m using an old Raspberry Pi Zero (not even W). If you’re starting from scratch, the Zero 2W looks like a better choice since it can do local wake word detection.

When I tried adding the Audio Enhancements I hit the following error:

Traceback (most recent call last):
File ".../wyoming-satellite-master/script/run", line 12, in
subprocess.check_call([context.env_exe, "-m", "wyoming_satellite"] + sys.argv[1:])
File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.../wyoming-satellite-master/.venv/bin/python3', '-m', 'wyoming_satellite', '--name', 'Wyoming Satellite', '--uri', 'tcp://0.0.0.0:10700', '--mic-command', 'arecord -r 16000 -c 1 -f S16_LE -t raw -D default:CARD=Device_1', '--snd-command', 'aplay -r 22050 -c 1 -f S16_LE -t raw', '--wake-uri', 'tcp://...:10400', '--wake-word-name', 'ok_nabu', '--mic-auto-gain', '10', '--mic-noise-suppression', '2']' died with ...

Seems that the binaries in the webrtc-noise-gain package aren’t compatible (at least with the Pi Zero). The solution was to build them which takes a rather long time…

sudo apt-get install python3-dev
. .venv/bin/activate
pip3 install –no-binary ‘:all:’ webrtc-noise-gain==1.2.3

It took a few hours but it works!

Play around with the audio enhancements settings to see what works for you.

I suggest adding an automation to pause the player as soon as the wake word is detected.

Posted on

What to do about that crappy dm9601 you got on eBay

Note that the title is past tense. Don’t buy these dreadful things intentionally!

They’re typically advertised as USB 2.0 10/100 Ethernet dongles. The photo will suggest that there’s a Realtek chip inside. It’s all a lie. They’re actually USB 1.1 and 10mbps half duplex. Speeds top out at under 8mbps (bits, not bytes). Pretty much useless for anything these days.

But let’s say you bought one. Worse yet, you bought one a few years ago, forgot how bad it was and bought another one.

Maybe your old one was hanging off a Raspberry Pi Zero running piCorePlayer and was plenty fast for local streaming even FLAC audio. Of course your other players have better network dongles (even the one that was salvaged from an ancient Tivo that was gathering dust in your part pile but I digress).

So you get the bright idea to salvage this useless dongle you just bought and swap it for one of the better dongles being “wasted” on one of your other players. It frees up the better dongle to use for the current project and the piCorePlayer will be just fine… or so you’d think.

You make the swap and suddenly both piCorePlayers with the dm9601 stop working! Swap back to the good dongle, reboot both, all good. Swap back to the dm9601 and both players go crazy. The music server shows both players winking on and off.

No… it can’t be. Do both dongles, purchased years apart, have the same MAC address? Your router seems to think so. You Google the MAC address (00:e0:4c:53:44:58) and start finding lots of confused and unhappy dm9601 owners. Pretty much all the threads ended in abuse (you idiot, just return it!) and frustration.

Ok. Yeah, that hypothetical sucker was me. Both times. I started searching for a solution and finally found this thread that seemed to reach a workable solution: https://forums.raspberrypi.com/viewtopic.php?t=167249.

I tried using ip (or in piCorePlayer’s case ifconfig) to change the MAC address. It worked but the change was only semi-persistent. It seems to generally survive reboots but unsurprisingly gets reverted on power off.

I then tried adding ifconfig eth0 hw ether "00:11:22:33:44:55"; killall -USR1 udhcpc to various places in the startup script but it wasn’t working consistently. Too early and it wouldn’t take effect. Too late and the player would come up with the default MAC and then change later leaving behind phantom players in the media server.

The working solution was to add the command to the udev rules to change the IP right as the dongle comes up – I didn’t even need to poke udhcpc afterward.
I created a file called 77-mac-fix.rules in /etc/udev/rules.d with the following:

ACTION=="add", SUBSYSTEM=="net", DEVPATH=="/devices/platform/soc/20980000.usb/usb1/1-1/1-1.4/1-1.4:1.0/net/eth0", ATTR{address}=="00:e0:4c:53:44:58", RUN+="/sbin/ifconfig eth0 hw ether 00:11:22:33:44:55"

For piCorePlayer I also had to add the file path etc/udev/rules.d/77-mac-fix.rules to /opt/.filetool.lst and run pcp backup to make the change permanent.

A word about that DEVPATH. You’ll need to find one for your specific dongle. Look in /sys/class/net/ to find the path for your dongle:

$ ls -l /sys/class/net/
total 0
lrwxrwxrwx 1 root root 0 May 30 11:45 eth0 -> ../../devices/platform/soc/20980000.usb/usb1/1-1/1-1.4/1-1.4:1.0/net/eth0/
lrwxrwxrwx 1 root root 0 Dec 31 1969 lo -> ../../devices/virtual/net/lo/

So for my device (eth0) the DEVPATH was /devices/platform/soc/20980000.usb/usb1/1-1/1-1.4/1-1.4:1.0/net/eth0. My two dongles were slightly different so yours probably will be too.

I hardcoded different MAC addresses for each of my players and now they’re both working as expected.

But really. These things never should have been produced in the first place. Don’t buy one.

Posted on

New life for old toys

I’ve accumulated a number of old PCs, laptops, and even a couple of servers over the years. With the recent chip shortages and supply chain problems I’ve taken to finding new applications for them.

Compatibility is surprisingly good. The x86-64 platform has been around for a long time. One problem I’ve hit recently is that UEFI has been around long enough so older BIOS’es can no longer boot some software which is distributed as disk images.

I ran into this with the HassIO “appliance” version of HomeAssistant. I was looking to set it up on an old ThinkPad T500 which didn’t have UEFI support.

But first a little digression.

Playing around with old hardware meant I was going to be trying all kinds of old installation disks. Either burning CD/DVDs or struggling with things like Rufus, UNetbootin (which I think is pronounced you-not-booting), etc. It gets old pretty fast.

Until I found a GPL-licensed tool called Ventoy.

Ventoy is a really cool tool. You install it on a flash drive with a simple command and from that point forward, all you have to do to boot an img or iso file is to copy it to the flash drive. You have store many images on the flash drive and you get a menu from which you can select which image to boot.

So far, I’ve used it successfully with Ubuntu, ParrotOS, Debian, GParted, and a few of the Puppy Linux images. The only one that gave me trouble so far is TinyCore Linux which seemed to have some UEFI issue – I’m not interested enough right now to troubleshoot it.

Anyway, this one’s a keeper. Check it out at https://www.ventoy.net.

I’ll have to get back to the UEFI boot story later. In the meantime, here are instructions for HassIO if you need them: https://community.home-assistant.io/t/install-ha-on-old-laptop-without-uefi/407443/20.

Posted on

Lazy Admin’s Guide To Changing Mongo Oplog Size

Have you read Mongo’s official guide to changing the size of your oplog http://docs.mongodb.org/manual/tutorial/change-oplog-size/ and found it a bit intimidating? Are you resizing it because you already have replication problems anyway? Might as well rebuild your secondary and increase the oplog size in one shot. It’s basically the same as the procedure outlined here: http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#automatically-sync-a-member.

1. Set the oplog size in /etc/mongodb.conf. Just add this line (the size is in MB):
oplogSize = 102400

2. Stop your server with:
db.shutdownServer()

3. Empty the data directory – the one set by dbPath. You can remove all the files but it’s probably a better idea to move them to a backup directory in case anything goes wrong.

4. Start your server.

That’s it. The server will find the empty data directory, initialize it using the new oplog size, rejoin the replica set and perform a complete initial sync.

Now, you wouldn’t want to do this on a huge production database but if your database isn’t too large, it’ll save you a bit of reading.

Posted on

More consistent iteration times

We were doing some quick and dirty load testing the other day, using a simple shell script to load messages into a queue in batches. The code looked something like:

while something ;do
echo Sending messages $(date)
send-messages
some-other-stuff
sleep 5
done

It seemed to be working but every few iterations, the time would skip by 6 seconds instead of 5. Obviously, the time it took to send the messages and do some other stuff was adding up. Since we’re in quick and dirty mode anyway, my first instinct was to run the send-messages asynchronously (the other stuff was printing log output and had to run in sequence), so we just added an & to the end of the send-messages line and the number of skips dropped by about a third.

This was an improvement but we were still skipping pretty often and we realized we could do even better.

Rather than using sleep to add a delay, we realized we could use it to act more like a timer. We started the sleep in the background at the top of the loop body and called wait once the body of the loop was done and ready to pause for the remaining time. It was a simple change:

while something ;do
sleep 5 &
slp=$!
echo Sending messages $(date)
send-messages
some-other-stuff
wait $slp
done

This kept each loop iteration really close to the 5 second goal. We might still see some drift over time but it was good enough for our purposes.

Posted on

WebSphere Jython scripting, add the script directory to the import path

If you’re running a script with wsadmin and it tries to import other modules that live in the same directory, you’ll discover another difference between Python and wsadmin. Python will always look for modules in the path that the script was run from. wsadmin won’t. This is kind of annoying if you have a few local import files. Sure, you can always add a directory to the import path via the command line but who ever remembers to do that (and who remembers the syntax)? Since we already have the script path (from fix5), we might as well add it to the import path.

if mainfile:
    mainpath = os.path.dirname(mainfile)
    if mainpath:
        sys.path[:0] = [ os.path.abspath(mainpath) ]

See fix5 for the computation of mainfile. Better yet, download the whole collection from here.

Posted on

Running the latest Bing on “not supported” devices

There’s a new version of Bing available here that supports speech input and turn-by-turn spoken directions. It even seems to have traffic avoidance. I’ve barely had a chance to play with it yet but I wanted to pass along a fix for the “Your device has been identified as not supported.” error you’ll get on any but the very latest phones.

Go to the Bing install directory, mine was “Storage CardProgram FilesBing”, and edit the file “Bing.config”. Find the entry “UseAppServerUpdates”. Change the value from “True” to “False”. Save the file.

That should do it. The application runs now and I don’t get the error anymore.

This was tested on an HTC Kaiser running some 6.5 ROM.