(Ashbysoft *)

County Holidays

Fri, 03 Apr 2026 08:13:15 +0000

For a number of years, Angela & Phil have been working through all the UK/Eire counties taking a holiday, this is where we have got to so far (check the article date ^^up there)

Map generated with the excellent MapChart.

SSO Yak Shaving

Mon, 02 Feb 2026 00:00:00 +0000

Yak shaving our SSO

Yes, that is a thing!

Well folks, the time had come to make a decent job of supporting our family on our homelab, and move away from big G. Having already sorted out basic observability with replicated logging and monitoring, plus cross site backup, it was time to put consistent user services in place.

Other articles will (probably) explain our setup for redundant email, calendar & contacts. We have alternate solutions for photos and music - Phil went minimal and just went for NAS shared to each player and added home brew face recognition. Stu was later to the party and went for immich & jellyfin respectively, (after using miniDLNA for several years.) Stu also has a home automation package called Domoticz which is quite widely used (just not as loudly trumpeted as Home Assistant!)

Of course, each service needs user authentication. To reduce friction with the family, we would really like to offer common credentials across all services, with single sign on for relevant services. Plus a password change UI. Hmm.

How hard can it be

We started with just chucking the relevant lines from /etc/passwd & /etc/shadow in git as we keep a shared repo in sync nightly between our two servers. This is ok for manual synchronisation, but leaves a potential big time gap in password sync if a user changes their credentials and neither of us are around to cut/paste the files! Also if we fat finger this operation we are in trouble… Finally, although our email software (courier-imap, exim4) can use OS authentication, other more 'cloudy' packages can’t access this so easily it seems.

So - time to put something more robust in place that we can leverage for all services…

Start from the OS level

PAM provides the Linux/Unix OS API for authentication, and there is good support for both LDAP (Lightweight Directory Access Protocol) and NIS (Network Information Service) back ends to provide centralised network data stores. (NIS was where I started back in about 1992 on HP-UX…) However, NIS is deprecated these days as unreliable and insecure, so… hello LDAP!

OpenLDAP FTW

OpenLDAP has been around for decades, and is a single binary (slapd) backed by a file store. So this is where we start our yak shaving… getting the package installed is trivial on both Debian and Ubuntu. Documentation is.. interesting! There has been quite an effort to update slapd recently, moving to an entirely new configuration structure stored in the database itself, and a new integration with PAM using an external daemon (nslcd). Great, if the documentation would ever catch up! Hmm. You can find more on OpenLDAP roadmap here and PAM+LDAP design here.

It took some time to find the right steps to get a base configuration that we could ingest data into from the shadow file; since we need a migration and we don’t have clear text password credentials for our users (rightly so!) Once the migration scripts were written/executed, and the source data filed away in git, we were able to remove the users from the /etc files and hey presto! We could still log in! PAM+LDAP provides a neat separation of system accounts (generally those below 1000, created by package installers and the like directly in the OS files) from our new, portable user accounts. Here’s what we ended up doing:

First, this line added to /etc/default/slapd ensures access via TCP, Unix socket & TLS/SSL

    SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///"

To update the user DB, we use the root Distinguished Name (rootDN) account,
which we created at install time. This cmd ingests an
LDAP Information File (.ldif) which has a standard syntax.

    # ldapadd -x -D 'cn=admin,dc=ashbysoft,dc=com' -y $rootpwd -f <filename>.ldif

-x tells ldapadd to use simple authentication which we need for rootDN access -D provides the rootDN value & -y the password, which we set at install time.

First we make an Organisational Unit (OU) to contain user accounts:

    # ldapadd -x -D 'cn=admin,dc=ashbysoft,dc=com' -y $rootpwd -f people.ldif

people.ldif contains:

    dn: ou=People,dc=ashbysoft,dc=com  
    objectClass: organizationalUnit  
    ou: People

We can then add our users. This is the .ldif template we used:

    dn: uid=${uid},ou=People,dc=ashbysoft,dc=com  
    uid: ${uid}  
    givenName: ${gn}  
    sn: ${sn}  
    cn: ${gn} ${sn}  
    displayName: ${gn} ${sn}  
    objectClass: person  
    objectClass: posixAccount  
    objectClass: inetOrgPerson  
    userPassword: {CRYPT}${pwdh}  
    loginShell: /bin/bash  
    uidNumber: ${uidn}  
    gidNumber: ${gidn}  
    homeDirectory: /home/${uid}  
    mail: ${uid}@ashbysoft.com  
    gecos: ${gn} ${sn}

where the parameters are taken from the /etc files. uid=Linux username, uidn=Linux ID. Ditto for group.
gn=given name, sn=surname, cn=common name. Adjust as required.
pwdh=password hash string from /etc/shadow file.
IMPORTANT: we prepend the literal text ‘{CRYPT}’ on the password hash field!

NB: We had already done the donkey work of syncing user and group IDs across our two servers - we were lucky to have very few clashes! Retro fitting common user credentials to disparate systems is always going to be messy… :)

Whilst getting setup, we decided to create an administrator user with permission to edit records. This avoids having to know or share knowledge of the rootDN password (that gives unlimited access to the data) with any client applications. We can then replicate the admin user's credentials. We also installed SSL/TLS certificates to provide secure access to clients if they needed it, using the system default ‘snakeoil’ keys :) This requires us to edit the configuration:

To modify the configuration database we cannot use rootDN access.
We need this magic cmd, run as the root user:

    # ldapmodify -Y EXTERNAL -H ldapi:/// -f <filename>.ldif

-Y EXTERNAL tells slapd to trust the OS user ID.
UID 1 has ‘special’ access granted in the default installation of slapd
-H ldapi:/// tells ldapmodify to connect using the default Unix socket: /var/run/slapd/ldapi
This enables slapd to locate the UID of the calling process

With this we can adjust the Access Control List data for our regular user ‘admin’.
admin-acl.ldif contains:

    dn: olcDatabase={1}mdb,cn=config  
    changetype: modify  
    delete: olcAccess  
    -  
    add: olcAccess  
    olcAccess: to attrs=userPassword by dn.exact="uid=admin,ou=people,dc=ashbysoft,dc=com" manage by self write by anonymous auth by * none  
    -  
    add: olcAccess  
    olcAccess: to attrs=shadowLastChange by dn.exact="uid=admin,ou=people,dc=ashbysoft,dc=com" manage by self write by * read  
    -  
    add: olcAccess  
    olcAccess: to * by dn.exact="uid=admin,ou=people,dc=ashbysoft,dc=com" manage by * read

dn=config database section to update, - is required between commands.
This ACL provides access to change the userPassword field for ‘admin’ and self,
to check authentication by the anonymous user and to block any other access.
(self allows updates by the authenticated user to their own records)
Note it must also provide similar access to shadowLastChange field and
provides update access to ‘admin’ and read access to anyone for all other fields.

We found the config database was divided up into several sections with different DNs.
you should check your database names are correct and adjust accordingly.
You can use this command to dump the config:

    # ldapsearch -Y EXTERNAL -H ldapi:/// -b 'cn=config' -s one -LLL -Q

This will dump the config database to stdout. You can ignore the ‘schema’ elements,
just look for the olcDatabase elements.

Exercise for the reader: can you find the magic ACL that allows root access to this data?

Here is the TLS/SSL certificate.ldif config file we used:

    dn: cn=config  
    add: olcTLSCACertificateFile  
    olcTLSCACertificateFile: /etc/ldap/ca-certificates.crt  
    -  
    add: olcTLSCertificateKeyFile  
    olcTLSCertificateKeyFile: /etc/ldap/ssl-cert-snakeoil.key  
    -  
    add: olcTLSCertificateFile  
    olcTLSCertificateFile: /etc/ldap/ssl-cert-snakeoil.pem

You will be getting familiar with this syntax by now!
This config adds the default system ‘snakeoil’ certificate & key.

Can I change my password?

We then needed to check what happened when a user executes the ‘passwd’ utility.. it works! Well, yes and no. The LDAP stored credentials did update, but they also became hashed with a different algorithm - one very old and from the distant past which is NOT secure :( It turns out there is basically a fight between which components create the hash; the LDAP client software, or the LDAP server itself. There is a 'new' method defined as part of an LDAP extension RFC 3062 that allows the LDAP server to create the hash according to a 'policy' set in the config. It turns out this policy is just two values (hash algorithm=CRYPT which directs slapd to the OS libcrypt() library, and salt format="$6$%.6s" which tells libcrypt to select SHA512 and a 6-digit salt), but they are stored in different database sections - sigh. The good news is that nslcd, part of the latest PAM+LDAP integration package (as of Q1 2026) already supports this method and infact does NOT support any other!

here is the crypt.ldif file we used:

    dn: cn=config   
    changetype: modify  
    add: olcPasswordCryptSaltFormat  
    olcPasswordCryptSaltFormat: "$6$%.6s"  
       
    dn: olcDatabase={-1}frontend,cn=config   
    changetype: modify  
    add: olcPasswordHash  
    olcPasswordHash: {CRYPT}

IMPORTANT! Note the hash directive is in frontend, whilst the salt is in the root section. No idea why…

With this setup, we get reasonably secure crypto hashing of passwords. BUT the shadow file was using a “$y$…” salt format. This is a reference to yescrypt which is a very modern, secure algorithm, so why did we not set this up? Unfortunately, we cannot select this as yescrypt is not supported by some client tools yet. These tools do not understand what is in the password field and fail to display, check or update the password correctly. Notable among those tools is phpLDAPadmin, possibly the most widely used FOSS LDAP management tool out there; boo! So SHA512CRYPT with a decent salt it is.. We have checked that such hashes can be exported back into a shadow passwd file and still work, (as they should since this was the default for Debian systems before yescrypt was introduced,) which is one of our requirements - to be able to back out a user credential to the file-based PAM system at any time, just incase (!)

Can I have a self-service web UI with that?

OK, so the user can change their password from the shell prompt. But let's be honest, how many will really do that? We need a web UI for this. So began many hours of hunting around for LDAP client UI tools that supported this need.. and we finally stumbled onto the LDAP Tools Project. This is a fantastic resource that happens to include exactly what we need - a basic web UI for user password change. Even better, they make this available as either a Debian package (from their own repository) or a container image from docker.io. Nice :) Since we are at the 'throwing at the wall and seeing if it sticks' stage, we installed by container image. Now, had we done this after the discovery of the password hashing conundrum, it would have saved a lot of time! As it was, the tool pretty much immediately worked, but left some wierd password hash that was nothing like the shadow file syntax and would not work there. Here began the journey of discovery about the aforementioned RFC 3062 and allowing the server to do the hashing. The default settings in this client are to create the hash itself (which uses whatever libcrypt version and configuration is in the container image, not the OS one) and upload the string directly to the LDAP field. Oops. Fortunately, the tool has a config value the lets it use RFC 3062. Win!

here is the tweaked config for this tool to match our ACLs and crypt setup:

    $ldap_url = "ldaps://<server address>";  
    putenv("LDAPTLS_REQCERT=allow");  
    putenv("LDAPTLS_CACERT=/etc/ssl/certs/ca-certificates.crt");  
    # use anonymous bind to check pwd  
    $ldap_binddn = null;  
    $who_change_password = "user";  
    $ldap_base = "dc=ashbysoft,dc=com";  
    $ldap_login_attribute = "uid";  
    # use RFC3062 EXOP to make LDAP server do the encryption  
    $ldap_use_exop_passwd = true;

Note we needed to allow self-signed certificates via LDAPTLS_REQCERT=allow
We do not need to use rootDN or even admin user access to check passwords
just anonymous and we can use ‘user’ to update the password via RFC3062 method.

Can I have an admin UI tool with that?

In addition to finding this great set of user tools, we came across a number of other LDAP 'admin' tools and we thought it would be prudent to have one of these handy as cryptic cmdline tools are an extra cognitive load during any debugging/emergency situation later.. we can recommend phpLDAPadmin however be aware of it’s limitation with yescrypt support. Otherwise it seems excellent, and is again available as Debian package or container image. We have the container version running right now..

Can we synchronise two servers?

Now, all this is very nice on one server, but what happens if it goes offline? How do our users log in on the other server to continue reading their email if the LDAP service is down? This is where we need replication, so we can run an LDAP server local to each site and keep them in sync as the data is updated.

Let's use OpenLDAP sync…

Those of you more familiar with OpenLDAP will perhaps know it already comes with replication tools… and indeed supports N-way synchronisation between 2 or more servers. Great! We tried to set this up; I’m sure it is possible somehow but we really struggled with the complexity of it. First it won’t work without SSL/TLS connections. Then it needs some Access Control List setup to allow servers to talk to each other. Then it needs a cron / systemd timer job to invoke the sync, or separate processes monitoring the DB for changes, then there is a queue of transactions to manage in a file, then… wow. All with cryptic command line tools. The list went on. Well done if you have set this up successfully on your site - we can see that for 1000's of accounts and 10's of servers it may prove worth the effort to understand and maintain. We didn’t bother; we felt this was too complex to manage and remember what to do in an emergency. It is also very opaque to monitor.

No :(

Let's use Syncthing instead…

We have already got reliable, cross-site file replication working nicely thanks to syncthing; a simple to manage tool available in most distributions with a simple to manage replication setup and decent web UI. Can we leverage this for LDAP sync? Why, yes of course! If one takes a close look at the built-in synchronisation, really all it is doing is exporting records when they change, queueing them for delivery to the other sites and importing them when they arrive. There is a lot of extra complexity in trying to export the minimal amount of changes and account for network failures, slower delivery, etc. There is also a big warning about 'split brain' situations where changes can occur in multiple servers to the same records at the same time (e.g. during a network outage) and resolving this later is a nightmare. Our situation is much simpler; we only expect the password field to change during self-service requests. This is the only thing we need to sync up. Any other change is manual administrator work and we can replicate it ourselves by whatever means we choose. So with this in mind, we can approach this in a very crude way:

run strictly in master/slave to avoid split brain. Only change passwords on the master (by only running the self-service web UI tool there.)
run a sync on a timer at the master - we chose 10 minutes as a reasonable time to replicate a password change. If the system fails in this window, the users will already know about it and can repeat the change action themselves if they wish on the recovered system.
export all the password hash fields at once - there are a few 10's of user accounts. No point being smart here, just dump them all out to temporary files.
identify each file with user’s DN from LDAP. This is guaranteed to be unique as it is also the database key.
check if each temporary file is different from the one currently being replicated - if yes, update the replicated copy. Then delete all the temporaries.
let syncthing resolve the replication across sites with version monitoring, queuing, recovery, etc.
run a monitor script at the receiver that checks each replicated file timestamp every few seconds. The number of files is tiny, so again no point being smart. Just check all timestamps
import the password hash field if a file updates.
to bootstrap the system, simply run the dump to make a initial dataset, then start the updater which will read the initial timestamps to monitor.
to restart either end, we just need a replicated file set locally to compare against. This should always be there once bootstrapped.
to force an update, simply 'touch' a file in the replicated set and it will be updated in the receiver.
if a user is added, their file will appear in the replicated set when dumped. New files will be imported at the receiver then monitored.
if a user is removed, their replicated file can simply be deleted and the receiver script restarted to remove the in-memory state.

Note that we do not have to care about the direction of sync with syncthing - it is N-way capable already. We simply either run the sync dump or the update scripts as required for master or slave status of the site. Having the replicated copies locally on the machine provides the startup state we need. This solution does depend on the system clock NOT going backwards between sync runs… but then lots of stuff will likely also break if the system clock is out by that much!

Yes!

here is the dump script:

    #!/bin/sh  
    # script to dump user passwords from LDAP and update any changed data to a target folder for sync  
    target="/var/lib/ldapsync"  
    tmpbase="/tmp/ldapsync"  
    tmpf="$tmpbase/$$"  
    admindn="cn=admin,dc=ashbysoft,dc=com"  
    adminpwd="/etc/ldap/rootDN.pwd"  
       
    # make sure only root can rwx our files..  
    umask 0077  
       
    [ ! -d "$target" ]  && echo "target $target missing, exiting">&2 && exit 1  
    echo "mkdir -p $tmpf"  
    mkdir -p "$tmpf"  
    cd "$tmpf"  
    ldapsearch -H ldapi:/// -x -D $admindn -y $adminpwd -b 'ou=people,dc=ashbysoft,dc=com' -s one -LLL userPassword |awk -v RS= '{print >("user" NR ".ldif")}'  
    for f in user*.ldif; do o=$(awk -F: '/dn:/ {print $2}' $f |tr -d ' '); mv "$f" "$o"; done  
    for f in uid*; do cmp -s "$f" "$target/$f" && continue; echo "update $f"; cp "$f" "$target"; done  
    echo "rm -r $tmpf"  
    rm -r "$tmpf"

Note we keep the admin credentials in a separate file,
we use ldapsearch to extract the userPassword field only
and feed it to a funky awk script that saves the output to separate files.
We then rename the files using their dn= line (more funky awk!)
compare them to the previous files and update any changed ones. /var/lib/ldapsync is the folder syncthing is sharing.

and here is the update script:

    #!/bin/bash
    # get a list of file timestamps and update userPassword for each one that has changed
    target="/var/lib/ldapsync"
    delay=10
    debug=""
    admindn="cn=admin,dc=ashbysoft,dc=com"
    adminpwd="/etc/ldap/rootDN.pwd"

    [ ! -d "$target" ] && echo "$target folder missing, exit 1" >&2 && exit 1

    cd "$target"
    echo "process target: $(pwd)"
    echo "initialise time stamps"
    declare -A ts
    for f in uid*
    do
        ts[$f]=$(stat -c '%Y' $f)
        [ -n "$debug" ] && echo "$f: ${ts[$f]} "
    done

    # loop to wait for ts updates
    # timestamp must move forward to update - old files are ignored
    # ts[$f] returns "" for new files so they will be processed and added to ts
    # removing files prevents further update but does not clear ts
    echo "wait for updates; delay: $delay"
    while true
    do
        sleep $delay
        for f in uid*
        do
	        t=$(stat -c '%Y' $f)
	        [ -n "$debug" ] && echo "$f: ${ts[$f]}->$t"
	        [[ "${ts[$f]}" -eq "$t" ]] && continue
	        # ts changed, update
	        ts[$f]=$t
	        echo " update: $f->$t"
	        # grok userPassword field and update LDAP
	        # dn: must occur first in LDIF, userPassword may be followed by continuation lines with ' ' first char
	        # also awk does not support filenames with '=' char so use cat |awk here!
	        pwdupdate=$(cat "$f" |awk '/dn:/ {inpwd=0; print $0"\nchangetype: modify"}
			        /userPassword:/ {inpwd=1; print "replace: userPassword\n"$0; next}
			        /^ / {if (inpwd==1) print $0}
			        !/^ / {inpwd=0}')
                [ -n "$debug" ] && echo "$pwdupdate"
                echo "$pwdupdate" | ldapmodify -H ldapi:/// -x -D $admindn -y $adminpwd
        done
    done

Mmm some more funky awk hacking for you :)
This awk groks the dn and userPassword fields from the source file
and removes anything else as we don’t want to update random stuff.
It outputs an .ldif that modifies the existing user record.
Why the inpwd stuff?
This is because the .ldif format enforces continuation lines
after 72 chars and we have to track that keep the lines we need.
Nasty. Hence the debug lines!!

I suggest this is still easier and more transparent than OpenLDAP sync…

Can we do SSO now?

No. LDAP provides a single reference source of authentication credentials and other user metadata to authentication tools like PAM. It does not provide a transferrable 'token' that can be used to log in to multiple services. That needs another technology layer…

eMail/calendar/contacts - LDAP only

Some applications do not really support the idea of SSO, like a single email account. The access protocols here are IMAP and SMTP, both of which require a username/password credential. There is no 'token' style login, so LDAP has provided us a single point of control for the password but not single sign-on. Calendar and contacts are similar if they use the standardised WebDAV based solutions of CalDAV and CardDAV.

Our chosen email IMAP solution is courier-imap which operates using the OS authentication and home directories to locate the users Maildir folder. Similarly, our chosen email SMTP solution is exim4 which also authenticates users using OS authentication and home directory via PAM to locate any .forward files and to deliver email files to their Maildir. Our chosen CalDAV/CardDAV solution is radicale through which we delegate authentication to IMAP.

This is the best we can do with eMail protocol level access that supports dedicated eMail client software like Thunderbird, Evolution, etc. We might be able to do better with a webmail client.. maybe. This is perhaps also why GMail desktop services are web based!

jellyfin part 1 - LDAP

jellyfin comes with an LDAP plugin built in. Yaay! Atfer some poking about, we made this work and could provide authentication externally to the jellyfin user database, so that was nice. But it's not SSO. And we still need to make accounts in jellyfin first, then set their authentication check to LDAP…

immich part 1 - No…

immich comes with many plugins - none of which do LDAP authentication :(

Domoticz part 1 - No…

Domoticz does not come with any authentication plugins. It also does not use OS authentication :(

OK so how do we do SSO?

Single what now?

Single sign-on requires a token that can be trusted and passed around to different services as an indicator that the user has 'logged on' to a realm and can therefore be allowed access to all services that are also registered with the same realm. One technology layer that does this is Open Authorization, specifically version 2.0. OAuth2.0 supports a range of use cases and deliberately does not specify how client software can register with a realm nor exchange keys with the same. There is a layer above OAuth2.0 called Open ID Connect that provides usage profiles of OAuth2.0 and extends the specification so client software can interact with realm software and discover how to manage the user and how to check their token.

Clearly the client software is our service application, like jellyfin, immich and Domoticz.

So, what is this magic realm software? You may find it under a few different names:

OpenID Provider (OP)
Identity Provider (IdP)
Identity and Access Management (IAM) <– you've probably seen this one from AWS :)
Authorization server <– this is the OAuth terminology
OIDC server
OAuth server

…blah blah blah. There seems to be a lot of twaddle and marketing-speak around this stuff as it's all big corporates and big money stuff doing security!

We have evaluated a few software packages that can provide this service - we have (so far) picked Keycloak as the most reasonable looking balance of: it working without huge bootstrap pain, being too opaque to understand / too badly documented to configure in a few hours, too huge and cumbersome for our little servers to handle it or need all the things it does. Also it must be able to reference/import user credentials from LDAP and provide full OIDC, not just OAuth2.0.

We looked at:

TinyAuth - this is not an IAM and does not do OIDC. It might fit your need for login controls for a basic web site.
Authelia - this might be an IAM. When we had spent the best part of 3 hrs trying to get it even to start up, we moved on. Shame, it looked quite efficient!
Authentik - this is a whopper! It worked, but we struggled to get it to load users from LDAP. It sucked up resources. It's also quite commercial in $$$.
Keycloak - this is the best balance we think. I’d expected a JVM in a container to be slow as h*ll, but it’s really not! Also memory footprint is OK. UI is clean!

immich part 2 - OIDC

So.. onwards with Keycloak, in a container deployment again. There are guide websites to setup Authentik with immich; after a bit more yak shaving we got keycloak to connect up and do the token thing by following the Authentik guide and adjusting for keycloak’s terminology/UI shifts. Yaay! We also got it to import LDAP users after poking it in the right place (apparently this is 'user federation' these days…) and hooray! It can issue tokens from the LDAP credentials!

So we are done, right? For immich - pretty much yes!

Just a tweak to configure immich to allow new users automatically and that’s it. Once the users exist in immich you can share content/etc. as normal. Logout is clean and directs the user back to the Keycloak logout page. Great!

jellyfin part 2 - OIDC

More yak shaving. Jellyfin has a plugin that supports OIDC/OAuth2.0. It is 'alpha' by declaration of the owner/developer. It is however used in every example I have seen of how to enable OIDC for jellyfin! I guess it is the only one out there… surprisingly, it does work. The config UI is awful and cannot even show you the previous values you have entered. So you have to retype ALL the data every time you tweak something. Nice.

BUT… it works! Great! You have to make your own HTML/CSS login button by abusing the ‘branding’ area of the login screen :) Logout does not work; it returns to the jellyfin login screen but you can press ’login with keycloak’ and it goes straight back in. No realm logout is invoked. So I made a new ’logout from keycloak’ button on the login screen by abusing the branding HTML area a bit more… It's working OK. It creates new users automatically. Can I remember the options I chose when I entered the form data? NO :( Hope I wrote it down… (I did.)

Domoticz part 2 - OIDC Error…

Oh boy… (!) Domoticz have a funny 'take' on OIDC. Their wiki says they support it and they describe how to configure a client app to access the server, but they kinda don't say only if you use the Domoticz server as the IAM as well. They solved a different problem - how to enable Android and iOS phone apps to do OIDC authentication so they do not store passwords out there in the big, wide world. Seems a good starting place for using tokens, and credit for going the OIDC route. Shame it is unusable regard using external IAMs (yet) because it makes several assumptions about what must be included in the token…

We tried, for many hours, to understand the code paths in Domoticz. We can see it has a parser for JWT tokens which we know Keycloak issues; this follows the OIDC standard. We can also see it checks the token signature against a key stored in the Domoticz database, which is good! We managed to get the key installed from Keycloak and it worked! Yaay! But now the token is rejected because Domoticz uses the JWT claim fields (OAuth has wierd names for stuff) for unusual purposes. Vis:

aud string must contain one value and this must be the client_id which we can store in the Domoticz DB (as a username!) This is OK! We can do this!
signature must be PS256 algorithm - we can do this too, and we can import the public key into the client_id user record in the Domoticz DB. OK!
issuer URL must be “https://$hostname:$port/” of the Domoticz server. Can't use external IAM then? OOPS :(
subject string must be a valid username stored in the Domticz DB. Keycloak passes UUID values, not under our control. We can't do this. OOPS :(
key_id string must be an ASCII number that matches the username record in the Domoticz DB. This is NOT how key_id works in OIDC! OOPS :(

For these reasons, OIDC will currently not work with Domoticz and an external IAM. I have no doubt their own IAM can make a JWT token like they expected. This is no use to man or beast as Domoticz cannot import users from LDAP and we cannot shove in the password hashes ourselves as Domoticz uses MD5 hashing which we don't. Plus we want SSO, not just password changes.

No SSO for you:(

Domoticz part 3 - OAuth Proxy FTW

We do not give up easily. Domoticz supports other means to provide external credentials, notably it supports Basic Authorization (and ancient X.509 Authorization) headers. Nice! These accept a username and clear text password that must match the Domoticz DB user data. Awesome!

So.. is there a way to provide a web front-end that can handle OIDC SSO and generate an Authorization header from it, then proxy requests from our browser? You bet there is! Say hello to Oauth2 Proxy - the weapon of choice for adding OIDC capabilities to legacy apps!

This little beauty is available as a pre-built Go binary or container (you know which I chose by now…) and acts as an OIDC client to login to the realm and request the magic JWT token. Once done, it passes the token back to the web browser in a cookie and becomes a HTTP proxy. Now every time a request flows through, it checks the token is valid, does a renewal with the IAM if required, and passes the request on to the upstream web server you have protected. Nice. If the token expires or is not renewed, boom! 401 unauthorised for you! Job done, right? Yes and no…

Remember we need a Basic Authorization header. This needs a username and password that matches Domoticz. Well, it turns out that oauth2-proxy can generate just such a header for us. Great! Turns out it also uses the JWT subject claim field for the username. Not great. This is that pesky UUID we saw before. However, there is also an option to prefer the email string as the username - now we're talking :) We can configure users in Domoticz with email addresses for usernames. Done right? Not quite - we still need a password. Now, given we have done a real user login in the oauth2-proxy to get the JWT token, we can trust the username is legit. The password serves no purpose anymore as we have already authenticated this username. So we can make them ALL THE SAME and use a static string here to just get by the Domoticz password check logic. OK? Turns out oauth2-proxy has a configuration to do exactly that already! I love this little guy :)

So that is what we do now: create new users in Domoticz with email usernames and the same fixed string password. Configure the password in oauth2-proxy and send this in the Authorization header. Bingo! It WORKS! Hooray! Here is our oauth2-proxy config snippet:

    pass_basic_auth="true"
    set_basic_auth="true"
    basic_auth_password="xxxxxxxxxxxxx"
    prefer_email_to_user="true"
    # to use this from nginx SSL proxy we need this:
    reverse_proxy="true"

Domoticz part 4 - Logout how?

So login works fine, but we press Logout in the Domoticz and as if by magic, we go straight back to the Dashboard page! This is to be expected; the Domoticz service has no idea we are connecting via the oauth2-proxy and thus we remain logged in, sending the Authorization header. How do we log out of this thing?

The guide has some instructions for us: first we call the oauth2-proxy endpoint to remove the session, and this can redirect the browser to the Keycloak logout endpoint for us. Nice! BUT… Domoticz web UI is all JavaScript and API calls, not web pages at all! What to do?

More yak shaving followed - we tried to block the API calls only; this results in a screen saying 'Domoticz is Offline' which is hilarious, but useless. We tried to log out of Keycloak, but the proxy just carries on working until the JWT expires and won't refresh. Then we noticed Domoticz has a Custom HTML template facility we can abuse… hehe. A bit like the branding HTML area abuse in jellyfin, we can add a URL button to the Custom menu in Domoticz that calls the required logout URL and we’re done! Hooray!

Here is the logout URL from Domoticz:

    https://<oauth2 server>/oauth2/sign_out?rd=http%3A%2F%2F<keycloak server>%2Frealms%2Fashbysoft-realm%2Fprotocol%2Fopenid-connect%2Flogout

Note the URLencoding of the redirect URL to direct the browser to keycloak logout screen after oauth2-proxy logout.

Domoticz part 5 - restore the Nginx SSL Proxy

You thought we were done here :)

Some things are left to tidy up. Unlike all our other services which reside behind our Wireguard VPN and do NOT use SSL (why bother for LAN only?) Domoticz needs to be exposed to the Public Internet. This is so we can get to it without fussing about with VPNs… and so our phone apps work.

Prior to implementing SSO, we exposed the Domoticz port via an Nginx proxy that terminated the SSL connection and forwarded requests. This we continue to do for the legacy phone apps that cannot handle SSO. For interest and amusement, we would like to expose the SSO web UI as a public end point. We may consider adding other services to the public rosta later…

So - oauth2-proxy comes with some guidance how to setup nginx to do the donkey work of the proxying, checking with the oauth-proxy endpoint if the request should be passed on or not. This is done using the auth_request module in nginx, which makes a sub request using the cookie from the browser and gets either 2xx or 401 status code in reply. We need to configure the oauth2-proxy to return our Authorization header in the auth check response, and we include this in the proxy request to Domoticz as before. We also need to adjust the redirect URL to return the browser to the public address, not the internal one when logging in. This means adding the URL to Keycloak and the whitelist in oauth2 also. Seems to work well…

BUT we can make it more efficient now we have a better proxy in nginx! If we review the Domoticz code paths, we see that only the API endpoint is checked for authorization. So we can replicate this in nginx with two location blocks, one for “/json” which catches the API calls, and one for “/” which passes all others. This is great BUT - now we get no redirect to login when we open the home page! So we need a third location header which catches ONLY the home page and redirects to the login endpoint for us. Did you know you can have location =/ in nginx which matches exactly the / request and nothing else? Cool!

..and that's how you shave the SSO yak. Well done if you made it this far!

PVCS Tools

Wed, 11 Jun 2025 20:00:00 +0000

PVCS Tools

TL;DR: I have just created and published an open source library + command line interface (CLI) to extract information and file content from PVCS log files - if this interests you, the code is on sourcehut as usual: https://git.sr.ht/~phlash/pvcstools

So err why?

If you remember the early days of revision control software on MS-DOS in the late 80s -> early 90s, you might remember the delightful Polytron Version Control System, aka Polytron VCS, aka PVCS.

PVCS was a popular commercial tool that provided similar capabilities to RCS (which was open source and ~1 year older by release date, but very much Unix / Posix based, so not available as a port to MS-DOS).

I used it both in work at BT for my day job, and at home to manage my own stuff. So far, so dull…

At the monthly tech meetup I attend (aka “nerd beers”), a friend reminded me of some toy software we had created that linked message boards together across Netware file servers (the message board package was Threadz Pinboard), to create a message “inter-network”. Later that evening I had a rummage in my archives and found the code - great, I can now pop that on a 3.5" floppy and present my friend with it next month as a surprise / late present from 1992 :)

Now the yak shaving began… the code was managed using PVCS but there were no checked out source files, all I had were the control files, aka log files, in their proprietary binary format. No problem I thought, I’ll go grab an old archive of pvcs stuff from the Inter-webs and extract the source - err nope, it turns out PVCS still exists and is now an expensive (£1500/user) corporate service platform owned / sold by Microfocus! Surely there is something from the past on archive.org I can use.. also nope (but see later). Oh well, it’s a 40 year-old file format, there will be open source tooling for that right… also nope!

So here I am needing to prank an old friend but without the tools to do it.. what’s an old hacker to do?

Reversing the format

I bite and decide to write some tooling that can extract source code from the log files. I have several hundred samples to look at locally, some large and with lots of changes, others smaller and with only one revision in, time to dump some hex..

It turns out that PVCS is very similar to RCS in keeping the latest file content, plus a series of reverse deltas to recontruct earlier revisions, it also supports symbolic names / labels (we might say tags in a modern VCS). There is a list of revision locks and some control metadata (such as the original file name, the originating user, etc.). All of this is visible in a logfile as plain text blocks, with some surrounding binary goo. Closer examination between files shows consistent structure, with a fixed header, then a series of Type-Length-Value (TLV) blocks, much like ASN.1 or Protobuf encoding. I work out how to step through these blocks, and to identify the one holding the latest revision of the content, extracting that to stdout. Job done for the prank, but…

The one that got away

While looking for PVCS downloads in the wayback machine, I discover that Borland C/C++ V5 for Windows supported PVCS natively as a built-in version control backend, and there is a CD image on archive.org :) Yoink!

After much unpacking (nested 3 layers deep - thanks Borland), I find a specific win32 DLL that contains the PVCS logic, which appears to have been supplied by Polytron to Borland (according to the copyright and version table strings within). I throw this into Ghidra and take a quick look - it’s quite complicated, so I decide to put that aside and see what I can learn from the raw files first, which turns out to be all I needed to extract source code - I might return to the decompiler if I ever want / need to know more.

Making a useful tool

I’m enjoying the process of understaning the format, there are likely to be others out there facing a similar issue (always assume another nerd shares your pain) and there are no open source tools for PVCS. I feel obliged to finish the job and extend the tooling such that it is possible to read useful metadata and retrieve any revision from a logfile, then publish for the good of geekdom - hello!

What can be done

Read-only extraction of metadata and content revisions (by revision number or label). It’s unlikely ever to be possible to write a logfile as there is too much unknown stuff (although some persistence with the decompiler might get there).

Have fun and let me know if you use this!

Introduction to the Fediverse

Tue, 10 Jun 2025 19:00:00 +0000

So Phil, what’s this Fediverse thing?

I’ve been know to bang on about alternative social media for a couple of years now, in particular the #Fediverse - now I can annoy you with a really nice video introduction produced by the talented Elena Rossini, enjoy!:

Phil Ashby CV

Tue, 03 Jun 2025 17:00:00 +0000

Philip Ashby

Contact

Address

<on application>
Felixstowe
Suffolk.

Phone

Mobile: <on application>
Home: <on application>

Email

phil@ashbysoft.com

Socials

Mastodon: https://mastodon.me.uk/@phlash
Website: https://www.ashbysoft.com/

Hobbies

Amateur radio satellite (AMSAT) development with the FUNcube team.
Open source software maintainer (mostly my own toys!):
Sourcehut: https://sr.ht/~phlash
Github: https://github.com/phlash
Amateur geneology, collecting my family tree (currently ~900 people known, back to 1765).
Sailing, I’m a qualified dinghy sailor and competent crew member on a yacht, with occasional trips across the North Sea to Europe. Previously a speedboat owner (too old now!)
Music, I’m a self-taught multi-instrumentalist, sound engineer and ex recording studio owner (now replaced with a summer house since the children have left home!)

Education

Masters Degree in Electronics, York, 1989
A-levels in Mathematics, Further mathematics, Physics & Chemistry with S-level distinctions in Physics & Chemistry, Broxbourne Grammer, 1985

Certifications

Member of Institute of Engineering and Technology (MIET)
Member of British Computer Society (MBCS)
Associate Member of Chartered Institute of Information Security (AMCIISec)

Summary

Phil is an experienced technical architect; happiest when being the conduit between technologists, team leads, product owners, fellow architects, customers and leadership teams to guide effective technical evolution for a business. Phil equally enjoys discovering new ideas and spaces, generating prototypes, co-innovating with others¹, reading manuals², delivering service evolution and continuous learning. Phil always likes to work his way out of a role by spreading knowledge into teams. Phil believes trust matters and information silos are bad.

¹ Not a fan of the word ‘brainstorming’
² Has been known to read the GDPR!

Skill highlights

Architecture (C4 models, TOGAF, SDLCs & enterprise architectures, Cloud native engineering)
Communication (within and around delivery teams)
Reverse Engineering / Software archeology, for architecture discovery, security evaluation, occasionally source recovery!
Forward Engineering, preferably in C#, Java, Python or C, but not a technology fundamentalist :)
Operational Engineering, with cloud tooling (eg: ARM), operational visibility (eg: Datadog)
Security Engineering / Information assurance including risk management and technical security (ISO27001 & PCI-DSS compliance, security awareness training, previously a Certified Ethical Hacker)
Information technologies, particularly: Windows Server, Linux, containers & VMs, config management
Research and learning, especially in new & cross-boundary areas

Experience

2013-2020: Technical architect for GB Group, keeping the company technology evolving successfully as it grew from ~300 people in the UK to ~1100 people in 40 global locations via product expansion and aquisitions (I consulted into M&A team). I chaired the Design Authority Board as a core forum of information exchange, and decision authorisation across the business. I led the evolution of company platforms from isolated monoliths to interconnected services, allowing flexible service creation for target markets. I contributed to the migration of a service from un-scalable hardware encryption to Azure KeyVault. I discovered and documented historical systems on aquisition / retirement of owners.
2007-2013: Security analyst for BT, where I was part of a small hand-picked group of technology experts providing investigation, risk assessment and mitigation development for BT and NCSC/GCHQ to manage the (critical) national infrastucture risks of purchasing and deploying far-eastern telco equipment into the core BT network. I have been unable to mention this until Feb 2019, when the ex-technical director of NCSC published this blog (the paragraph below the final list of bullet points refers to us).
2000-2007: Consultant with IDL, providing various services to other organisations, ranging from product design and development (eg: mobile games, high availability routers) to customer communications (eg: BTexact website management and CMS).
1996-2000: Technical lead for BT Wireplay, where I led the development team creating the UK’s first (inter-)national computer gaming platform, provided the early architectural designs and enjoyed my only ‘Internet era startup’, from which I resigned after 12 months (it was already dead!)
1989-1996: Developer in BT, where I worked on both design and software development for automated speech services (eg: Callminder/1571), speech recognition technology (using neural networks) and gained valuble experience in operational services as Callminder went to production.

A Museum Piece

Fri, 25 Apr 2025 00:00:00 +0000

Well well, if I were to ask you to name the most famous inventors in the world of communications, who would you include? James Clerk Maxwell for his equations, Heinrich Rudolf Hertz for his demonstration of electromagnetic waves, or Guglielmo Marconi for his engineering to turn theory into practice and build the earliest working radio systems?

Here we have the Marconi RS-4 as we found it, missing a couple of knobs and looking well used.

Here it is with the case off, showing that the interior chassis paint seems quite good condition.

Here is the view inside the case, showing some dust and dirt however all the valves are in place and undamaged.

Here is a close-up of the tuning capacitor rack, showing a little surface corrosion but otherwise ok. The tuning dial still moves as well!

Here is a close-up of the RF section valves, showing a mix of Osram and Marconi brands. Most likely some have been replaced in service.

Inside the case we found a rather damp but intact circuit diagram!

The model label clearly shows the Type and a Serial number, along with the company name of Marconi Wireless Telegram Co. Ltd. London. Also an 'Instrument No'.

The radio is now in the Marconi Collection. More information here about the collection

Valve gear renovation

Fri, 25 Apr 2025 00:00:00 +0000

I have been restoring/renovating some olde worlde electronics recently, so I thought I should write a blog post about it (now Phil has explained how our new website works!)

This stems from the process of slowly clearing out my late Father-in-Law's farm sheds… We keep finding old, rusty things; some of which just scream at me to be repaired!

#1 The American Army Radio

Project #1 was a famous WWII radio receiver; the RCA AR88D, as seen in all good warzone bunkers from about 1942 onwards. This particular model is quite early and flaking with rust from the 50Kg steel shell. Yes, 50Kg! I can barely move it. This turns out to be a good thing as it appears to have saved all the valves from high-G shock loads which is generally what kills them.

Suffice to say, once the rusty case was levered open I found a remarkably well preserved chassis, with even some pencil marks to note servicing and repair work. Nice. At this point I have no idea if any of the electronic components are serviceable - I do the stupid thing and wire a plug onto it to try it - and surprise myself by it NOT emitting magic smoke. Infact the lamps come on! But nothing else…

So after surviving doing the one thing you should not do with old valve gear, it was time to get out the voltmeter and start checking values against the circuit diagram… Unsurprisingly many components had been damaged by decades of damp ingress.

Every single old oil filled paper+foil bypass capacitor was blown; leaky, unmeasurable capacity, etc. These 'bathtub' style components house 3 wound cylinders in a soldered metal can and are bolted to the chassis. I decided to replace the innards and keep the cans for a more authentic appearance. Chopping my way in and scooping out the innards was messy but satisfying :)

I also found there were capacitors marked 'Micamold' that were infact just paper and foil devices encased in bakelite. These also had all failed - cue more replacement work.

Finally a lot of the carbon film resistors had gone 'high' and also needed replacing to get the valve terminal voltages within spec. A few had blown entirely; mostly due to capacitor leakage (see above!)

On the upside, all the valve heater filaments seemed to be intact! This is extraordinary! I thank both the designers for specifying metal cased valves and the weight of the chassis in damping any possible shocks.. I was hopeful we had a full set of working valves :)

Also all the wound components seemed to have survived OK along with the oil-filled power supply capacitors that are much larger and tougher than the bathtub types…

Time to make a CPC order for a handful of new ceramic 630V capacitors and get soldering!

After a short wait and some painful soldering (access is NOT the best!) it was time to power it up again! Quite unexpectedly the damn thing actually WORKED! Testing alongside a 1990s Yaesu we find the frequency is ‘good enough’ on the manual dial and the sensitivity just as good as the modern receiver, after some gentle twiddling of the controls. Amazing bit of kit..

Some more pictures during the restoration..

As found condition

Inside - top

Inside - below

The tuner

A note

Cap innards

More info from the Radiomuseum here

#2 The Kitchen Radio

This is a pre-war Bakelite cased 'kitchen radio' with a glass backlit tuning scale. Station names from the past are inscribed in the paint across three bands: long, medium and short wave.

On releasing the rusted tight case screws (mind the Bakelite!) we find a seized tuning mechanism and a very rusted chassis. Several components have burst open from the damp ingress, and black dirt encrusts everything. The corroded valves are certainly 1930s era designs with top caps and octal bases. Several wires immediately break and fall off where the solder has corroded away and the copper gone green with verdigris.

The speaker is seized and rusty; no chance of this ever working again. All the components likely need replacing, and we’ve no idea if any valves may have survived, but it seems improbable :(

The value in these is mostly cosmetic - they were made cheaply and in large volumes, parts are hard to come by and there are no popular radio stations on these bands anymore!

So I decide this one is to become a display item with new innards. A visit to CeX returns a £20 Bluetooth speaker in a nice round tubular case. Much cleaning and a swift application of two cable ties later, and the new speaker is strapped just behind the rear grille, allowing one to still operate the buttons through the gaps. Nice. The paint on the back of the tuning glass needs some touching up with a matching colour.. try not to cover over the names!

Now we just attach a trailing USB charger cable, and we’re done!

Dirt & dust

Rusty chassis

Rear view

Broken valve

#3 The Party Piece

Here is a Pilot TRG108 portable gramaphone and radio unit. The autochanger's mechanical parts have been freed up and are working fine :) The audio amplifier section has been restored, however we await a solution to to the decayed cartridge innards before it plays records again..

Again this thing has no real value, but it's a fun toy to play our 78rpm records :) If I can get it going for little to no investment, I will. The unit uses a BSR8 record deck - very common and needles are still available at reasonable cost. The electronics use four valves; two for the radio section, one audio amplifier and one power supply rectifier. We need to be VERY CAREFUL as the chassis is wired directly to the mains plug (!!) thus saving costs by using an autotransformer for voltage changing. It also uses Uxx series valves which have a common 0.1A heater filament at various voltages. These run in series from 125V AC source. Hmm.

Unfortunately of the four valves installed, one was smashed and two have broken heater filaments :( I had a box of valves donated to me by a friend many years ago - mostly TV ones. By pure luck I found an equivalent for the AF valve (UCL82 -> PCL82) and a rectifier valve (EY85->EZ80) I could substitute! The main issue is the heater filaments are different current / voltage ratings.

So far I have managed to replace all the leaky capacitors, a few duff resistors, the rectifier and AF valves. The rectifier is a half wave originally (urgh!) so the full-wave replacement is connected with both diodes in parallel to ensure I retain enough current rating. Seems to work ok.. The AF valve is an equivalent so no hacking required :)

I have managed to swap around the autotransformer to obtain outputs close to the filament ratings I needed; by running the EZ80 filament off the 6.3v lamp output and the PCL82 between the 220/250 step down taps in series with one lamp, we have a solution!

I have disconnected the filaments & HT of the radio section. There is nothing to listen to on MW/LW anyway :) I may want to re-instate some radio function later.. for now we just abandon this part of the circuit.

Unbelievably, the speaker cone is NOT seized up however it rattles like a bag of spanners, so I swapped it for a Kenwood 4-inch car speaker I had lying around. I also wired in a 1/4" socket so I can use it as a guitar amp. Nice. That completes the work on the audio electronics, just the cartridge to go.

Unfortunately, after purchasing two replacement stylii, I quickly discovered the pick-up had zero output on either 78 or LP sides. So it was time to disassemble it and find out what was going on.. as you can see from the photos below, there is significant damage through damp; verdigris on the copper contact plates and it appears the crystal has dissolved & turned to mush! With no repair possible, I need to find a replacement cartridge…

I have picked up (haha!) a BSR X3M cartridge to swap in as the original mono BSR TC8M has very low vertical compliance and would damage stereo LP records if played. So it’s probably for the best it is knackered! Does anyone want a pair of stylus for the TC8?

(The replacement stylus is actually from another vintage player! A 1968 ITT Kolster-Brandes KP036 ‘Coloursound’ from a local auction. £10! This one is transistor based, so less interesting for me to fix up..)

As found..

Servicing!

Soldering!

What crystal?

The donor

More info from the Radiomuseum here

#4 The Ones That Got Away

Several other items have been salvaged from the scrap pile; notably two 1950s era top quality Hi-Fi systems and a car radio unit common in Jags at the time! These all went to happy enthusiasts on eBay for a respectable amount of cash :)

A Radiomobile model 4010

Crusty radio

Insides..

This unit is 6V positive earth, a variation of the more widely known model 100 used in upmarket 1950s Jaguar models. It includes a ‘vibrator PSU’ - essentially a mechanical oscillator, transformer and rectifier valve to provide HT. There is considerable rust and decay of the outside, but inside is not so bad! Even the push buttons kinda work.. but not the tuning. The control is seized and the cable snapped immediately on removing the case!

I have no use at all for such a radio given it would only work in ancient cars.. so off it went to a Jaguar restorer for parts.

An RCA New Orthophonic High Fidelity system

Fancy HiFi in 1950s

Rusty chassis

Expensive valves

This system is a classic of it’s time, considered an equal to Quad and similar high-end manufacturers. We have here boxed pre-amp and tuner head units, and an unboxed power amp. The system is mono; stereo came later! As you can see, the power amp chassis has suffered a lot of surface rusting. Underneath the components are all splitting from damp ingress and the cables are rotten. The head units fared much better; protected by the wooden box there is little decay and clean insides. The unit runs a pair of KT66 output valves in class ‘B’ giving a comfortable 12 watts r.m.s.

I decided I did not really need a mono amp, even one as rare and nice as this could be! Also I had no way to really sort out the surface rust and decay to make it look nice - most of the appeal today is cosmetic. So off it went to a Hi-Fi restorer who could give it some love :)

More info from the Radiomuseum here

A Pye Mozart stereo Hi-Fi system

Shiny Pye

Filthy amp

More $$$ valves

This Pye HFS-20M system is surprisingly a similar vintage to the mono RCA above, but here we have a smaller footprint and a more modern appearance. The unit is less powerful, and runs a pair of EL34 output valves in class ‘A’ to obtain 2 x 4 watts r.m.s. stereo. (A mono unit was also sold.) The head unit is again quite clean inside having been boxed, whilst the power amp is filthy; thick with dirt and rust. I didn’t spend any time checking if this could be restored as it’s only desirable in good cosmetic condition! It is however, complete and valves appear undamaged.

Off it went to the same buyer as the RCA system above :)

More info from the Radiomuseum here

About (Ashbysoft *)

Thu, 24 Apr 2025 08:36:00 +0000

We are a small family software firm with a wide range of experience in embedded systems, personal computer applications, and distributed systems fields.

People..

Phil Ashby: Previously a technical architect with GB Group. Member of the FUNcube satellite development team.
Stuart Ashby: previously a systems architect with NDS/Synamedia, a major technology provider to several large TV companies.

Experience..

We consider ourselves to be expert in both high level OO design (systems and software), and low level systems programming with published work in both areas, including:

a real-time interactive TV shopping platform (QVC Active on UK Cable TV)
a UK scale interactive speech system (CallMinder)
a world-wide multi-player gaming system (Wireplay3)
a Windows device driver (Wireplay.VxD)
an Internet content filter (ICRAfilter)
a professional hosting service portal (BT Selfcare)
the on-board computer system for several satellites (FUNcube)

We enjoy working with the C/C++, Java & .NET ecosystems, where our depth of experience allows us to work with many target environments (eg: Windows, Linux, VxWorks, Solaris, OpenTV), in particular hetrogeneous distributed platforms.

We also have some experience with assembly code on Intel x86, SPARC & ARM CPUs, but prefer not to mention it in these 4th-gen language days ;-)

Happy Customers..

We have sucessfully delivered software for large and small customers such as:

AMSAT, where Phil is a member of the FUNcube development team.
broadcast TV software/security system migrations for Sky Germany, Sky Italy, Astro Malaysia, KBW, beIN
broadcast TV feature upgrades including PVR, IPTV for Sky Mexico, DirecTV, Sky UK, OSN, Airtel
QVC UK, whose QVC Active shopping system was developed by Stuart and his team in his day job..
MTV, whose interactive service on Telewest is powered by software developed by Stuart in his day job..
BTexact, whose Internet site content is managed and published through a system developed by Phil in his day job..
The Internet Content Rating Association, who asked Phil to develop their network filtering software (ICRAfilter).
Wireplay, where Phil & Stuart lead the development team whilst working for both BT and Gameplay.com.

Contact Us

All enquiries for more information, or quotes: enquiries@ashbysoft.com

Support requests for existing customers: support@ashbysoft.com

Phlash - the career bits

Sat, 29 Mar 2025 15:30:00 +0000

About Phlash - the techie career bits

[this post has been migrated back here from https://dev.to where I am no longer active]

I have been coding for 40+ years. I’m a recently retired (yay!)[edit: now part-time employed] senior technical architect (sounds great, means: do all the stuff nobody else wants to, make hard decisions and take responsibility for everything we ship!).

At a young age I made myself a promise: the day I get up and don’t want to play with a computer, I’ll give up this hobby and get a proper job…

A brief career history:

BT PLC from 1989-1996 as part of the speech application division (yep, SAD team), where I helped create CallMinder (now 1571) and early telephone banking systems, while learning all about speech recognition technology (neural nets, now known as machine learning or ‘AI’ :grin:)
Wireplay / Gameplay from 1996-2000 where I was technical lead for the UK’s first online (well, dialup!) video gaming service, and had my first unicorn startup experience, including resigning after 12months to…
Internet Designers Ltd from 2000-2007, an IT consultancy that grew from the initial 20 or so people to ~110 at its peak, where I worked on a multitude of smaller projects that varied from high reliability routers, through Java mobile games to managing BTexact Technologies website in a custom CMS..
Back to BT PLC, from 2007-2013 to become a security analyst and hand-picked member of a specialist team investigating the risks and developing the mitigations while BT deployed £6Bn worth of far-eastern telco equipment into the UK’s core communications network. No biggie!
GBGroup PLC from 2013-2020 when I retired. I was their first ’technical architect’, and helped design and deliver their identity intelligence services as they grew from ~300 people in a couple of UK offices to a global multi-national with 1100 people, 40 offices round the globe and a multitude of products and services, many a result of mergers and acquisitions (I also consulted into the M&A team - enlightening!)
WTW from 2024-present, where I am working 3 days a week to support their architecture function and generally be a nuisance!

I mostly program in these languages: C#, Java, Python, C, shell, ASM & enthusiasm!

If you like what I do, feel free to buy me a beer 🍺

EMFcamp 2024

Thu, 20 Jun 2024 18:00:00 +0000

EMFcamp? What’s that Phil?

Oh boy.. you are one of today’s lucky 10,000!

Probably best to start with the EMFcamp website, although that only really scratches the surface of what happens when 3000+ geeky / nerdy (why not both) folks get together in a really rather nice field (thanks Eastnor Deer Park!) and let their inner geek out.

I have already read through a few blogs from other attendees that describe the experience from their point of view and it’s interesting that some things appear quite different to people, while others are very firmly the same - I’ll start by trying to pick out the commonality:

For first timers, an overwhelming sense of having “found my people” - it’s quite beautiful to watch. My first time (only 2 years ago) was a revelation - despite being one of the lucky ones who already had some geeky friends and family (yo bro!), being with so many other geeks was a delight.
The joy of being able to be yourself, all of yourself and be accepted even congratulated on this.
Knowing that you share a special trait with your fellow attendees: curiosity is our hallmark and it’s encouraged! You just know it’s fine to pop into a village and ask about their stuff / clothes / challenges making things, or go to one of 400+ talks / workshops / events. A wealth of willing information firehoses to drink from..
Wonder (awe even) at the ability of others to create / repair / restore / reuse / innovate with things.
The obvious amount of self / social capital (I don’t like that term but it fits) invested by the team of people who volunteer for months ahead of the event to make EMF happen, the willingness of attendees to volunteer during the event, paying forward to ensure we and others get this experience again :)
The diversity of attendees in many forms.
The willingness of people to play, especially regarding silly things like a ‘Cat ears must be worn beyond this point’ sign, that was honoured by many.

Whew! Perhaps you were expecting a list of fun techie things, or shared camping experiences, but it seems it’s the social aspects of EMF that everyone gets.

What was it like for Phil?

Speaking of camping, I did suffer from the cold overnight - my feet did not like me much (cramp!) and I’m wondering how to keep them warmer without leaving a heated blanket on (bad idea!). There is a campervan option that my friends took (expensive on your own), or even staying in a local cottage / AirBnB / hotel (then you miss out on the nightime entertainment however).

I popped into the Null Sector for a few minutes to experience what has elsewhere been described as “filthy” bass (I estimate several kW of large bass bins stacked under the DJ gantry), and can confirm that I was moved (at least my eyes went out of focus) by much of the infrasound being generated! This however isn’t really my thing (never has been) so I retreated to a safe distance and enjoyed the laser show from both Null Sector and the projected video games on the hillside (cheers Seb!). Walking back from Null Sector I realised that some of the lasers (above the entrance) were controlled from a keyboard 50m away in the field which was there for us to play with - at that time a couple of pre-teens were doing a good job staying with the beat..

As a member of the AMSAT village, I spent half my time talking to others about satellite technology: building them, getting them into space and using them from the ground. Lots of interest, both in how and why we do this, which was great!

This time I volunteered for a shift in the shop (lovely people, mostly working terminals) and as a herald on stage introducing speakers (more lovely people, opportunity for terrible jokes / puns), so you might see me briefly on the recordings of some of the talks (Stage C, Sat afternoon). The free meal in the volunteer kitchen was excellent too - thanks team!

Finally - I decided to spend as much time meeting people as I could, since I could catch up on talks later, which was definitely worth it for me - the Mastodon meet-up was good (they’re all real people!), the craic in the Robot Arms (bar tent) was chill, I especially liked the waves of applause as / when someone beat the high score on the games around the sides, the beer was decent and service was prompt. Old friends were there from 2022 and earlier, and I talked one of my oldest friends (from uni nearly 40 years ago) into coming too.

I’ll be back #emfcamp.

Scheduled Hugo Posts

Mon, 01 Apr 2024 00:00:00 +0000

This post was written earlier..

It should appear automatically on the blog in the early hours of April 1st..

The mechanism

As per all scheduled things in Linux / Unix, this uses cron to run the blog publishing script every morning at 02:30hrs. Content is processed by Hugo taking into account the date: header in metadata, and not pushed to the public site until we are past that point. Simples.

Wireplay calling - hello 1996!

Mon, 11 Dec 2023 23:45:41 +0000

UPDATE: 2023 - We got interviewed

Recently, the excellent CrossedWires Podcast asked us for an interview, how could we refuse? You can use the previous link to listen to us steamrollering poor James the host!

UPDATE: 2022 - Off to the museum

All the physical goodies below have now been donated to the Centre for Computing History in Cambridge, UK. We hope that in time they will be merged into their exhibits on on-line gaming and multi-player games.

It’s amazing the things you find in people’s lofts, in this case a pristine 1996 BT Wireplay trial pack, let’s see what you got way back when..

This is a CD sized stiff plastic case, about 3cm thick.. inside we have:

A sheet of instructions, the unopened feedback questionnaire, and a multi-CD case. Here’s what the instructions say:

We also have a covering letter from the operations manager (Hi Richard!):

Opening the CD case we find more paperwork at the front, inlcuding some sweet untouched stickers!

..then at the back, the CD with the client software:

So you want a copy of the CD contents right… Come Get Some! NB: this ZIP contains both the trial (V1.4A) and the later updated (V1.5H) client.

Bonus finds!

Stu also had the install CD for the first V2.0 Windows & DOS clients:

Phil just found this in his garage, one of the CD-RWs created to send to the pressing plant (this is probably an escrow copy, archive no 2):

So do they still run?

Of course they do :)

anyone fancy setting up a server for these?

Downgrading WordPress after an upgrade incident

Wed, 22 Nov 2023 14:07:16 +0000

OK Phil, what did you do this time?

Me??! Nothing…
…
OK, so a friend runs a collection of websites, which she generates offline using WordPress and the excellent WP2Static plugin, allowing us to host the sites behind a CDN (Azure in case you’re curious), and without having to manage WP online. This is officially a 2-way win for cost and security.

To do this, I built her a container image using a variant of Lokl that I have added an Azure uploader into, and shrunk the image size as far as possible, including replacing MariaDB with SQLite. She thus runs n (currently 4) instances of this container on her own machine, and changes are automatically synced to the hosting service in real time.

Earlier today she was looking at the admin panel and noticed that WordPress had an upgrade available, with a nice attractive button… sometimes you forget that you aren’t looking after an online WP instance, so the old “always upgrade for security” reaction kicks in. Suddenly she’s looking at this:

..and WordPress doesn’t work anymore. Oops.

Better call… Phil?

Because she’s a smart cookie, this is where she stopped poking and contacted me - correct!

Now it’s possible that we could complete the upgrade, but it provides no useful new features for her sites, and doing this previously (out of curiosity) broke a few bits inside Lokl and WP2Static that took a considerable amount of time to find and fix. Our preferred solution is therefore to revert the button click and downgrade WordPress (remember, this is all offline so there are no security issues with old WP code running!). So err, how does one do that?

What did the Upgrade button do?

Because she runs >1 instance of this container, we can take a look at the installed WP files and find out what changed on the affected instance: pretty much all the .php files in the WP installation folder are newer, there are new themes and a couple of plugins have appeared.

Luckily all the actual site content in wp-content/database and wp-content/uploads seems unaltered from the last set of edits made (a few days ago).

The downgrade process

Once we had determined that the database and media files were still intact then it was clear that we could pull the WP installation from another unaffected instance and unpack it on the broken one, tar to the rescue!

Here’s the specific workflow:

[host] Connect to the broken container: docker exec -it <broken> /bin/bash
[broken] Back up the site content: cd <wp install>; tar czf /tmp/content.tgz wp-content
[host] Connect to a donor instance: docker exec -it <donor> /bin/bash
[donor] Yank WordPress install: cd <wp install>; tar czf /tmp/wp.tgz .
[host] Transfer the donated install: docker cp <donor>:/tmp/wp.tgz - |docker cp - <broken>:/tmp/wp.tgz
[donor] Tidy up donor: rm /tmp/wp.tgz; exit
[broken] Stop WP (YMMV): ps |fgrep php; kill <php-fpm pid>
[broken] Nuke WP install: cd ..; rm -rf <wp install>
[broken] Unpack donated WP: mkdir <wp install>; cd <wp-install>; tar xvf /tmp/wp.tgz
[broken] Remove content: rm -rf wp-content/database wp-content/uploads
[broken] Unpack backed up content: mkdir restore; cd restore; tar xvf /tmp/content.tgz
[broken] Restore database: mv restore/wp-content/database wp-content
[broken] Restore media files: mv restore/wp-content/uploads wp-content
[broken] Tidy up broken: rm -rf restore /tmp/content.tgz /tmp/wp.tgz; exit
[host] Restart: docker stop <broken>; docker start <broken>

Now go check stuff is working :)

Sip tea in relaxed manner.

Face Recognition

Tue, 19 Sep 2023 12:23:34 +0000

TL;DR: Where’s the code?

Hi impatient person, it’s here: https://sr.ht/~phlash/Face_recognition

Motivation

Google killed off Picasa desktop, which Phil has been using for a number of years, in particular because of it’s ability to recognise faces and tag / group them. With over 20000 photos, and 200+ people tagged, it’s time to get out before losing all that knowledge.

Data escape

Provided the appropriate option is ticked, Picasa will save a .picasa.ini file in each folder it knows about, containing rectangles that locate faces in image files, and where those faces have been tagged, the name tag used. While this preserves the output data, it does not include the trained recognition model Picasa is using to accurately match faces, this is buried in a custom binary database, and unlikely to make any sense outside of the particular recognition engine used, so we need to start again for this…

New recogniser tech

After a bit of rummaging Phil found the excellent face_recognition project from Adam Geitgey: https://github.com/ageitgey/face_recognition and his clear blog series explaining stuff: https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78, fabulous job Adam!

Armed with this code, and after a fair amount of trial & error getting dependencies working (see below), some testing showed that careful tuning of image size and algorithm selection could result in decent face detection & landmark extraction through a batch of images without eating vast compute time.

Sane storage

Because I intend to save overnight image processing results in a format that can be easily queried by other tools on multiple systems, I need portability, indexing and structure, preferably without a lot more installation / maintenance pain: step forward SQLite :)

The current database schema supports two distinct jobs:

- keeping a record of all files (images or not) in the searched areas, detecting changes and duplicates (via file hashes). - keeping a record of facial co-ordinates in each image (file hash), grouping them into similar clusters and labelling clusters.

Bootstrapping the clustering

Since I already have tags from Picasa, on some image files, I can pre-load clusters from this Picasa data (including the multiple groups per label), which gives me a starting point for clustering unknown faces too.

Sage advice

Having got this far, I ran off the end of the tutorials from Adam and was unsure on how to approach the classification / training stages, so I asked my friend and colleague, GBG’s resident data scientist Ian. After my poor problem description, Ian wisely suggested that I probably didn’t have enough training data to get a good machine learning model, it may well ‘overfit’ and have problems, besides that’s all hard work with new libraries (Scikit probably), and there are effective simpler means to an end: analysis!

Encoding analysis

Ian suggested I try and improve the basic face matcher that comes in the face_recognition package, by looking at the variance of the elements in a facial encoding (128 of them) across my data set, and weighting towards those with higher variance when doing simple vector (euclidean) distance measures. He also thought it would speed up the comparison if I reduced the n(n-1) complexity back to n by averaging the already labelled/grouped Picasa faces and comparing new data to those averages, rather than every member of a group. Good advice, thanks Ian!

gThumb as a viewer

I went looking for a way to visualise outputs as it’s tedious picking through log files and firing up eog for sample images. After some initial rummaging at https://alternativeto.net I installed gThumb from the standard distro and started work on a plugin https://git.sr.ht/~phlash/gthumb_faces, which currently intercepts image rendering to draw facial rectangles on images, while also extending the tree list with a Faces entry showing all facial labels.

face_recognition install

Adam provides instructions on installing the C library used (dlib) from source here: https://gist.github.com/ageitgey/629d75c1baac34dfa5ca2a1928a7aeaf

Having been through this, I discovered that the python package manager does just as well, so my final install list to get this working was:

# aptitude install python-pip python-setuptools python-numpy libboost-python-dev libjpeg-dev cmake
# pip install dlib
# pip install face_recognition

Speeding it all up

With everything in place and mostly working, I started running lots of image files through.. which is when I discovered the need to tune face detection settings to get any workable speed. Unfortunately the current tuning misses a lot of faces that aren’t square-on (it relies on HOG first, only running CNN if nothing turns up, or not at all). I need GPU power to make this all go faster!

Fortunately Simon has just upgraded our desktop PC which means there is a spare AMD Turks (6670) device and the Xeon / Haswell integrated Intel graphics device I could use.. Cue installation montage of AMD card into my server and BIOS poking to stop it disabling the Intel GPU when the AMD was plugged in :)

NB: If you want both integrated and extension GPUs working in a Dell machine - you must [enable ‘Multi-Display’ in the BIOS] (https://www.dell.com/community/PowerEdge-Hardware-General/Dell-PowerEdge-T20-xeon-Multi-Display-bios-setting/td-p/4542733) and you must have a VGA device (or emulator) plugged into the VGA port at boot - otherwise the BIOS turns off the integrated chip.

Now it gets fun again - I’ve never written any GPGPU code, and it looks like I may have to port the underlying dlib library from CUDA to OpenCL, since there is no CUDA for AMD or Intel devices (kind of..).

I thus started from zero: got a demo working, then got an FIR filter implementation working (thanks gr-clenabled) then started comparing performance between CPU, Xeon GPU, AMD GPU and was surprised that the Xeon GPU only just outperforms the CPU, while the AMD GPU is slower, due to data shifting delays across PCI. Hmmn. I’m still looking at tuning the code for AMD, since I was hoping for at least 10x over the CPU, perhaps an FIR filter isn’t a reasonable testbed?

A word of caution here: at one point I wrote some bad GPGPU code which didn’t specify resources correctly, and didn’t release them either - running this a few times started producing errors, then hard locked my whole server when it crashed on the Intel GPU - eek!

Obihai OBi110 Andrews & Arnold VoIP Setup

Wed, 21 Jun 2023 19:45:51 +0000

What’s this?

A little back story and terse summary of the settings I am using to enable VoIP (Voice over IP) telephony from my existing Obihai OBi110 bridge unit, so my analogue phones still work without an analogue line any more..

TL;DR? config below

The Obihai OBi110

PC-Magazine wrote a review back in 2012 when these were a shiny new toy explaining what they are for: bridging telephone technologies together.

My Current Use

I bought the OBi110 to use as a super configurable call screening device, with a built-in auto-assistant to inform callers that they are being screened and to press a specific DTMF key, or be diverted to a voicemail system on my local server (this bit never happened!). The call screening has been in place for several years, working well, and allowing me to block many nusiance calls.

The Plan

It’s simple - as the OBi110 supports VoIP via the SIP standard, and A&A can cost-effectively port my existing BT landline number to their SIP VoIP service, I just have to figure out the configuration required to make it work. Behind a Network Address Translation (NAT) router. When nobody has ever done this before with an OBi110 (at least not written it up!). Without b0rking my very well known home phone number that I have had for almost 35 years.

The detailed plan to protect my number!

Buy a temporary test number (they’re super cheap!).
Use that temporary number to get the OBi110 working with A&A SIP.
..or.. if that fails: buy a known-good SIP to Analogue Telephone Adapter (ATA). Get that working.
Once something is working, port my real number in and switch over to it.
Release the temporary number.

The side effects

Asking A&A to port my existing number in will have dramtic side effects on my landline services: as this is the only number, BT will assume I no longer want their telephony and issue a PSTN cease on my line, which will also go to my old broadband provider (Plusnet), who will immediately cease their service too. A non-reversible and final cessation of my landline and old broadband.

The Config

By section, top-to-bottom in the side menu. NB: only shows non-default settings

System Management, Network Settings (these are specific to my internal network - adjust for yours!):

Parameter	Value
AddressingType	Static
IPAddress	192.168.0.8
!SubnetMask	255.255.255.0
DefaultGateway	192.168.0.1
DNSServer1	192.168.0.5
NTPServer1	192.168.0.5
LocalTimeZone	GMT+0

System Management, Auto Provisioning:

Parameter	Value
ITSP Provisioning/Method	Disabled

System Management, Device Admin (I collect syslog on my server, you may not want to):

Parameter	Value
AdminPassword	…
UserPassword	…
Syslog/Server	192.168.0.5
Syslog/Level	7

Service Providers, ITSP Profile A, General:

Parameter	Value
Name	Andrews & Arnold
STUNEnable	Yes
STUNServer	stun.aa.net.uk

Service Providers, ITSP Profile A, SIP:

Parameter	Value
ProxyServer	voiceless.aa.net.uk
RegistrarServer	voiceless.aa.net.uk
UserAgentDomain	voiceless.aa.net.uk

NB: Setting a proxy server is against A&A recommendation but OBi110 considers a service disabled without it, it provides a default if RegistrarServer is not set.

Voice Services, SP1 Service (as above, I collect detailed syslogs, you may not want to):

Parameter	Value
X_RingProfile	B
X_SipDebugOption	Log All Except REGISTER Messages
AuthUserName	<allocated number>
AuthPassword	…

NB: Ring profile B is for UK ringing tones, which are less weird for us.

Voice Services, OBITALK Service:

Parameter	Value
Enable	No

Voice Services, Auto Attendant:

Parameter	Value
Enable	No

Physical Interfaces, PHONE Port:

Parameter	Value
OutboundCallRoute	`{([1-9]x?(Mpli)):pp},{(<#:>\|999):pli},{0:aa},{:aa2},{(<1:>(Msp1)):sp1},{(<2:>(Msp2)):sp2},{(<8:>(Mli)):li},{(<**9:>(Mpp)):pp},{(Mpli):pli}`
PrimaryLine	SP1
RingFrequency	25
CallerIDMethod	FSK(V.23)
CallerIDTrigger	After Polarity Reversal

NB: Outbound call route replaces 911 with 999 and routes to pli instead of li.
NBB: CallerID settings are for my UK, BT provided analogue phones. YMMV.

Physical Interfaces, LINE Port:

Parameter	Value
Enable	No

The Result

With the above settings, I can dial a standard number (no prefixes) and it calls out on the SP1 SIP service, connects and can send / receive audio. Incoming calls to the authenticated number are routed to the analogue phones, which correctly answer and can send / receive audio. Success!

NB: I can still force use of the PSTN line by using the **8 prefix in the outbound call route.

END.

Debian to Andrews & Arnold L2TP Setup

Fri, 09 Jun 2023 15:30:26 +0000

Why?

Because I’m changing my Internet Service Provider (ISP) from one who provides a static IP address (Plusnet) to one who doesn’t (Virgin Media)^ unless I buy a silly money business product, however I still need to provide various services to friends and family (email, chat etc.) which requires a static IP address.

Conveniently Andrews & Arnold (aka A&A, another ISP) sell a service specifically for this situation, where you get a static IP address in their network that can be associated with a network interface on a machine in your world over the Layer 2 Tunneling Protocol (L2TP).

The Plan

I currently have a static IP address on my home broadband. This allows me to forward specific traffic from the Internet to a server I run in the house, using my OpenWRT router. Easy(ish).

I will replace this with a remote interface (over L2TP), directly to the server, returning my router to a simpler configuration (almost no traffic forwarding), in preparation for replacing it with the Virgin Media one. I will also need to migrate the firewall rules from the router to the server to filter the raw Internet on this interface (but not any others). Sounds OKish..

Caveat: I presently have one forwarding rule for the L2TP traffic (port 1701/udp), which may not be necessary with sufficient tunnelled traffic or keep alives. To Be Experimented With.

Step #0 - buy the L2TP product

This was definitely the easy bit, A&A have a sane and working fully automated sales platform, so on a Sunday morning I completed my purchase in about 10 mins. I particularly like the way they integrate with their payment provider, automatically filling out my details once I have sent them a setup payment.

Step #1 - learn `iptables`

As I will be migrating firewall rules, and will want them installed before exposing my server to the raw Internet, I looked at this early.

I read the manual for ufw which is the usual firewall tooling recommended by Debian and Ubuntu. Unfortunately it’s designed for systems where you want to firewall everything, and arranging it to only mess with one interface is ugly. I was also unsure the integration into the networking control stack will work for me, I do not use network manager on my server.. next!

I looked at FirewallD which also seemed popular, unfortunately that integrates even more with network manager and is aimed at road warriors.. it may have managed the multi-homing better.. next!

Given I really don’t want anything ‘clever’, and I’ve managed to read and understand what fail2ban is already doing with the underlying iptables kernel configuration, I dove into writing my own ruleset and hooking that in to the INPUT processing. This pretty much worked first time, although I did have a minor struggle with unplanned additional traffic (see below)!

Here’s the final script (hooked into PPP, see below: /etc/ppp/ip-pre-up.d/firewall.sh):

#! /bin/sh
# iptables firewall setup script for L2TP tunnel interface exposed directly to the 'net

# some config..
IFACE=$1
CHAIN=aaisp-in
IPTABLES=/sbin/iptables
#IPTABLES=echo
# services we permit
SSHD="22/tcp"
SMTP="25,465,587/tcp"
IMAP="143,993/tcp"
HTTP="80,443/tcp"
TINC="655/tcp 655/udp"
MTRX="8448/tcp"
# munged together..
ALLOW="$SSHD $SMTP $IMAP $HTTP $TINC $MTRX"

# bail out function
fail() {
    echo "$*" >&2
    exit 1
}

[ -n "$IFACE" ] || fail "no interface on command line"

# first, clean up
# - disconnect INPUT chain from calling our user chain (for current interface)
echo "Dropping existing firewall rules.."
$IPTABLES -D INPUT -i $IFACE -j $CHAIN
sleep 1
# - nuke our user chain
$IPTABLES -F $CHAIN
$IPTABLES -X $CHAIN

# now build our user chain
echo "Creating firewall rules.."
$IPTABLES -N $CHAIN
for s in $ALLOW
do
    PROT=`echo $s |cut -d/ -f2`
    PORT=`echo $s |cut -d/ -f1`
    [ -n "$PORT" ] || fail "missing port number(s): $s"
    $IPTABLES -A $CHAIN -p $PROT -s 0/0 -d 0/0 -m multiport --dports $PORT -j ACCEPT
done
# permit established connections (that we have made from inside)
$IPTABLES -A $CHAIN -m conntrack --ctstate ESTABLISHED -j ACCEPT
# default log then reject after all allows
$IPTABLES -A $CHAIN -j LOG --log-prefix "_FIREWALL_ "
$IPTABLES -A $CHAIN -j REJECT

# connect current interface to user chain
$IPTABLES -A INPUT -i $IFACE -j $CHAIN

# done!
exit 0

As you can see, I run a few TCP services, plus a tinc VPN (TCP & UDP). The original plan was not to have the default route via the new interface, so I wouldn’t need to allow any other traffic apart from the well-known services. Unfortunately this didn’t work (routing issues), so I now include the additional ESTABLISHED rule which allows Internet things to reply to any packets my server sends out (like DNS!).

Currently I’m logging rejected packets, with a prefix that allows me to dump the logs into a separate file (/var/log/firewall.log). YMMV.

Step #2 - hook up the L2TP & PPP

A&A provide a guide for Debian which was quite helpful, but doesn’t really reflect my use case. I also read the manuals for the software used and it turns out there are better ways to acheive some things - might poke their wiki with an update at some point :)

required packages

# aptitude install xl2tpd pppd

static routes for the tunnel

This ensures that even if the default route changes (it will) then tunnel traffic always goes out over the underlying network (/etc/network/interfaces):

# The primary network interface
# - additional static routes for l2tp.aa.net.uk so it remains routed via gateway even
#   after xl2tpd/pppd brings up a new default route via the tunnel
auto eno1
iface eno1 inet static
        address 192.168.0.5
        broadcast 192.168.0.255
        netmask 255.255.255.0
        gateway 192.168.0.1
        up /sbin/ip route add 90.155.53.19/32 via 192.168.0.1
        up /sbin/ip route add 194.4.172.12/32 via 192.168.0.1

xl2tpd tunnel client

Here we provide the L2TP daemon with connection details for A&A, and a useful option that automatically starts connection attempts. We also ask it to retry effectively forever if the connection fails. This avoids the ugly cron job hack in the original guide to start the connection (/etc/xl2tpd/xl2tpd.conf):

[lac aaisp]
lns = l2tp.aa.net.uk
autodial = yes
redial = yes
redial timeout = 15
max redials = 9999
require authentication = no
pppoptfile = /etc/ppp/peers/lac.aaisp

Note that this needs working DNS to locate the tunnel server. At this point we have a default route to the Internet via the router, so we can resolve names (I happen to run a full recursive resolver on this system, which is paranoia on my part and you may be happy to simply forward requests to your ISP!)

pppd remote interface

Almost the final part of the puzzle, this configures pppd to negotiate an IP address and create a new network interface associated with it (/etc/ppp/peers/lac.aaisp):

# PPP outbound config for AAISP/L2TP tunnel
# - do not require peer to authenticate with us
# - replace default route via this interface when up
# - call the new interface 'aaisp-l2tp'
# - include our AAISP credentials
noauth
replacedefaultroute
defaultroute
ifname aaisp-l2tp
name <insert your A&A L2TP identity here>
password <insert your A&A L2TP password here>

Note that this replaces the default route on the server (previously to my gateway at 192.168.0.1) so that connecting clients get replies from the same IP they sent packets to. I tried configuring without adjusting the default route, but things get messy with trying to associate incoming and outgoing packets in the network core and force non-default specific routing.. Maybe later when I’ve read another manual! I also have other interfaces on this server..

pppd will automatically run the firewall setup script (see above) as this interface comes up, providing the interface name as the first argument. Note that as I only run one ppp connection, I don’t need to filter on interface name in the script to avoid firewalling other things. YMMV.

Fire up the beast!

This is the easy bit:

# systemctl start xl2tpd; tail -f /var/log/syslog

..and watch the magic happen :)

^ Also why: because they are the only alternative to a terrible aluminium POTS line that will never go faster than 50/1.5, and there is zero sign of Openreach putting actual fibre along my road. Le sigh..

Swingland

Fri, 12 May 2023 19:00:00 +0000

Irritation-driven-development

Back in the past I wrote a couple of Java apps. These were written using the Java Swing framework, and have worked nicely on both Windows and Linux X11 desktops for a long time.

More recently I switched away from X11 & Budgie to pure Wayland for my, desktop on the assumption that it’s over 10 years old now, and is the default technology underlying current Gnome and KDE desktops.. everything will be fine right? Kind of..

It turns out Java GUI technology has been somewhat abandoned by Oracle, in particular when running Swing (or JavaFX) applications on Wayland, Java relies on the XWayland adpater to bodge X11 API calls back to Wayland proper. This is not great (read: doesn’t work out of the box), so I wondered if someone had implemented a native Wayland backend for Swing - and they have, multiple times - super, but wait.. Oracle have not merged either project into mainline Openjdk, and indeed the only way to make either of them work is by hideous bodging, which also puts paid to any thoughts of writing my own backend, as the hacks required to load it at run time (read: in every application) are not sustainable. Grrr! So what to do?

I can of course switch back to an X11 desktop (ie: Budgie), but this irritates me, plus I have already ported OpenSceneGraph + Flightgear to Wayland, it wasn’t that difficult :)

Swing, like most medium-large frameworks has documented public API and internal implementations, so I could choose to re-write Swing, retaining the public API (thus existing applications can re-compile against my library) but rewrite Swing internals from the ground up on top of Wayland APIs. Sounds sensible right? However Swing (and the earlier AWT it relies on) have several hundred documented classes, with many more undocumented internal classes and a fair amount of native code (in C) to connect to X11, Win32 or Cocoa (macOS), plus there are no Java bindings to Wayland, only the libwayland C API. How brave do I feel?

Looking at my existing apps (see above), I have only used a small number of Swing features, so I should be able to create a spike that supports just the features I need to prove my proposal.. still seems pretty mad, but I have time (the glory of retirement!), so let’s go!

Wayland API in (mostly) Java

Given I will be using most of the Wayland protocol to achieve anything (it’s minimal) and Wayland is well specified then the ‘start at the bottom and build up’ design pattern fits. Wayland has a wire protocol based on a Unix socket, and usefully Java has supported Unix sockets since release 16, so I can write everything in Java to (de)serialise messages. This gets me going quickly, providing positive feedback that I’m on the right track..

Unfortunately Wayland also uses two Unix features that Java doesn’t have: SYS V shared memory buffers; file descriptor passing over Unix sockets. I then discover a bug (or misunderstand the documentation!)..

Shared memory solution

Helpfully Java Native Interface (JNI) does allow memory buffers to be passed from native code back to Java, and vice versa using DirectByteBuffer objects - so I only have to write enough JNI/C code to allocate and destroy shared memory, which I can then efficiently access from the Java code.

Passing file descriptors

This is a less obvious solution, since Java does not provide public access to the native file descritor(s) for any operating system APIs, however the source code is available identifying where a socket stores this native value (which appears to be stable over many releases), and from JNI/C I can easily access the required object field without the security challenges of doing so in managed Java code. I’ll live with this slightly hacky solution for now (until Java has accessors for native descriptors, it’s been proposed a few times!)

Checking for input

Wayland operates asynchronously between client and server, there are only a couple of occasions when a client needs to wait for input and we can simply block in a read call, at other times we need to poll for input using a check for data being available to read. This should work via the java.nio.channels.Selector mechanism, but sadly it does not. Thus I extend my JNI/C a little to provide a working data availability check, using the above hack to obtain the native file descriptor and calling ioctl(FIONREAD), which does work!

Testing ensues..

As soon as I have enough of the protocol implemented to connect and display something, then testing ensues - gotta see something on screen :) The test program will now evolve along with the protocol components as I add features. Oh, and it works nicely - this is 11 days in!

Swing API in many steps

With the underlying Wayland protocol up and running, I move on to a ‘baby steps’ exploration of the Swing API, working down from one of my simple applications (the powermonitor) and creating ‘just enough code to pass the tests’. This takes considerably longer than I hoped, as Swing is rather tightly coupled, and small refactorings (such as hoisting state to a super class) result in a lot of changes to coupled code - bah!

Along side the real application, I create a Swing test application which provides scope for short cuts and internal testing during development without hacking my real app about (always good to have something you can run after each change!)

After another 5 days work, I get to see the first Swing app run, with almost no useful features, but API compatible with the real thing. Neat!

Features, then more features

So from here I start picking off features to add and working through the stack to ‘make it so’ for each: first up a single 8x16 MONOSPACED font that I ~~steal~~ borrow from The Ultimate Old Skool PC Font Pack, the original IBM VGA text mode font. Retro computing FTW.

This is followed up with input handling: first keyboard then mouse, both rather buggy; then more high level components (Dialog, JLabel, JButton) over the next few days, followed by improvements to Graphics primitives.. and lots of debugging, particularly around the interaction with my tiling window manager (sway) and how it chooses to position new windows.

Cursor support bogs me down for a bit, especially the interaction between an asynchronous protocol, and the need to change the cursor on demand while the pointer is within a single Wayland window, also the increasingly complex logic required to flow input events through the component stack, which I refactor a couple of times.

Help arrives!

About a month into the project, my eldest son arrives for a family visit, and gets interested in the madness.. he pulls the repository, builds the code and it runs first time, on his Pinebook, which is both an ARM (not x86) and big-endian by default - I’m almost impressed this just works!

He also find and fixes a couple of protocol bugs for me - thanks Martin!

Dogfooding time

At this point I decide to write an app that both uses Swingland and supports it, a font editor for the custom file format I have created to store simple bitmap fonts - this allows me to create a proper cursor font without hand-editing in a hex editor (which I have been doing until now!)

The process of creating this real application finds more bugs (of course!)..

Image support

The next feature is the ImageIO library that allows bitmap images to be loaded. Because I have contributed to it and I think it’s cool, I start with the Quite OK Image format, then Portable Network Graphics as most images will be in PNGs. Transparency and alpha blending follow as both image formats support alpha channels.

Window manager interaction

Switching an application to full screen (and back) under program control is explicitly supported in Swing, and useful for games or similar, so this goes in next, and requires yet another refactor of the way Swing interacts with Wayland

Menus!

At this point we’re approaching the 80% useful mark of the Pareto Principle, Swingland is looking like an actually useful libray (tm), although it really needs application menus.. cue another round of refactoring, popup window handling and event processing re-design, and we have it! Looking good:

Getting braver with testing

Having exhausted my own applications as test beds, I decide to move on to the Oracle-provided Swing demo apps, in particular the TopLevelDemo and ButtonDemo ones that should now work with a simple change of imports and a re-compile.. well almost ;) More debugging and fiddling with the detail of button processing ensues.

More fonts..

The latest feature to go in is support for X11/PCF font files, which opens up a world of reasonable fonts for text. I had looked at TrueType (TTF) fonts earlier and decided that they were far too much work for now (a whole virtual machine and binary executable format to just fiddle with point positions during layout.. err nope!).

Ongoing development

I hope to maintain Swingland as an actually useful library for some time, and would particularly welcome other contributors who need other Swing features that they could develop. Come get some hot source

DOOM! on the EMF TiDAL badge

Fri, 22 Jul 2022 14:00:00 +0000

Does it run DOOM?

It’s becoming the defacto question for any piece of technology with a display on it.. fridges, printers, all sorts of things can run DOOM.

So why not this shiny new ESP32S3 based electronic badge I have just received from ElectroMagnetic Field 2022? It has a decent dual core 32bit Xtensa MCU, 8MB of RAM, 8MB of flash storage (partitioned up - see below) and an ST7789 colour pixel display.

Oh, and a joystick :)

Similar work & self-imposed constraints

There is a port of DOOM for an ESP32, proving that it can be done, however I have added additional constraints on myself as follows:

Co-exist with the Micropython environment that comes pre-installed on the badge.
Create an app that can be distributed from the TiDAL Hatchery

So what does the micropython runtime environment look like?

It’s based on the Espressif IoT Development Framework (IDF), which defines a storage partitioning scheme that I cannot replace (but I can use to my advantage..)
ESP-IDF, and thus Micropython are not designed as an OS with runtime application loading/unloading in mind..
Micropython is extensible at runtime, via pre-built .mpy format modules, which can be compiled from C.

..and the Hatchery format?

A folder named after the app (eg: /apps/Doom), that must be a python module (ie: contain an __init__.py file), plus any other files required to run.
The __init__.py must define an instance of an App class (or subclass) that the badge shell can locate / load.

Cunning plan #1, and failure.

How hard can it be to simply compile up DOOM as an .mpy extension, patch in the I/O (display, files, user-input) back to Micropython and load it up?

It turns out the Micropython extension format is very homebrew (for perfectly good reasons), and comes with significant limitations compared to an off-the-shelf application format such as ELF or COFF, in particular:

No declared data, there is no support for loading a .data segment as part of a module.
No static global variables (you know, the bedrock of most C programs!) as it cannot re-locate references when loading into memory, only visible globals can be used.

I also had to choose a starting version of DOOM, with an emphasis on portability. This turned out to be the easy bit, enter DOOM generic, which happily compiled and ran first time on my Linux desktop ;)

Oh, and of course I had to write a new scaler for the video to reduce 320x200 => 240x135.

The grind..

So, first remove all declared data from the DOOM sources, by moving initialisation into suitable xx_Init() functions if they are available, or adding them if not. ~2000 lines of code (LoC) changed, a number of ugly corner cases simply removed (like being able to change key bindings) and it runs on the desktop again - phew!

Now remove that static keyword from all the global variables, and prefix them all with a local module abbreviation to avoid collisions. Another ~2000 LoC changed, and it still runs on the desktop - double phew since this took several days!

Chuck all those changes into a large git patch file, and save it away. Done.

Tooling fail.

Compile everything according to the Micropython supplied tooling. Discover it cannot link certain object files - grrr :(

Fix Multiple, bugs in Micropython.

Chuck all the Micropython bug fixes into a git patch file and save that away. Done now?

Runtime fail.

Errr nope. It turns out Micropython makes more assumptions about how code is compiled, in particular that all things are relocatable in memory via a single Global Offset Table (GOT). DOOM makes different assumptions, and repeatedly crashes in unpredictable locations due to memory corruption.

After a few days trying to debug this fiasco (with printf of course since I cannot use a debugger due to proprietary module formatting and a special loader), I give up.

Cunning plan #2, and success (eventually!)

Back to basics

Having dismally failed to run anything so far, I decide to start at the other end, building up from a Hello Mum! program that relies only on the built-in ESP-ROM (and is compiled without the Micropython “special tooling” that is causing me problems), designed to run as the first user-provided code on the device, in ESP-IDF parlance, as the “secondary boot loader”. Thus is born phlashboot, the shortest program I can write that says ‘Hi’ without crashing.

After some trial & error with watchdog timers, this works - relief!

A new plan forms

Remember I mentioned a Partition table that ESP-IDF uses? This puts application code in a specific location within flash^ (one of two ‘over-the-air’ or OTA update partitions), along with a header that defines where in memory it runs, including if it’s memory mapped. The normal boot flow of an ESP-IDF application is that the secondary loader applies the defined memory mapping or copies code/data into RAM (hello .data!), then jumps to a defined entry point. Neat.

This is how Micropython itself get’s going, can I re-use the same flow to load/run my application at known memory locations outside Micropython, without having to use the broken .mpy module format? Baby steps ensue..

^ for those of you wondering why not memory map from the filesystem part of flash? The MMU is not granular enough for this, hence having to map from a specific ‘OTA’ partition offset.

Gently does it..

I adapt phlashboot so it doesn’t expect to be the bootloader, instead it looks like an OTA application, which the ESP-IDF secondary loader is happy to memory map (mmap) and run, next step..

After a great deal of reading through technical manuals and rummaging in the code, creating the romread test application and generally poking at (and crashing) my device, I understand how the memory manager works, what I can and cannot touch and in theory how I can memory map and execute another app within micropython. There is also a fair amount of investigation into the Micropython memory layout, see memstuff folder! Thus is born doomloader.mpy a short and separate Micropython module that pokes at the MMU to load an OTA app into Micropython-safe address space. It works :D, next step..

DOOM is back baby!

I can now use standard tooling, to build a standard application and run it on device “underneath” Micropython, so back to DOOM. There is some fiddling with compiler options and load addresses to get a sensible OTA application out, which fits easily into a flash (~460k, OTA partitions are 2MiB), the existing plan for connecting in I/O is still intact(!) so in theory it should work… and it doesn’t.

It turns out that DOOM expects a little more POSIX compliance from it’s C library than Micropython provides, in particular the printf implementation is lacking many features. The good part is that because I’m building an entirely separate binary application, I can use someone elses printf, and finally, after a lot of pain, it runs!

The colours are awful. I can’t actually control anything yet. I’m deliriously happy :D

Final fixes

The colour being mangled turns out to be an endian difference between the MPU and the display device, easily determined by looking at the code for the display driver, which byte-swaps everything when drawing into the device, unless it’s blitting a buffer, when it assumes it’s already swapped. Fixed.

Providing user input is a case of finding the right DOOM keys to fake when pressing joystick buttons or others on the badge.

Show me the code!

To horribly mis-quote another of my favourite games come get some baby!

..OR..

Grab the built objects from the hatchery

The required demo video..

Sorry, your browser doesn’t support HTML5 video

HDMI Switch Reversing

Mon, 14 Feb 2022 21:33:45 +0000

PortTa HDMI Switch 4:1 with Audio

Nearest archived product page I could find

Phil bought one of the above recently to avoid excessive grubbing about behind the TV changing over between XBox, Laptop(s), PS4, DVD player… you get the idea. It was cheap, it kind of worked. Unfortunately it had a couple of features that made it useless in my circumstances: no auto-switching to active input; overlapping IR codes with my Humax box (change channel on the Humax and the switch powers down!)

Needless to say I have had no joy asking the vendor for info on fixing these bugs, so I could send it back and try another brand, or.. open it up :) Turns out a couple of folks have taken a look inside similar devices, mostly to hack round HDCP protection, so I have some idea what to expect and indeed it’s pretty unexciting:

There’s a couple of devices from Explore Microelectronics that do the work (EP94A1K: HDMI switch, EPF021A: MCU), an SPI audio DAC to provide analogue outputs and some lights and buttons.

I grabbed the data sheets for the MCU device (or similar, as they’re closed source / NDA stuff) and after much Internet hunting also found some programming tools written in C#/.NET for XP that may work with the MCU (EPF021A).

The plan:

* activate the ICP (in-circuit programming) mode in the MCU
* extract the existing firmware, prove I can re-program the device
* put a toolchain together that can build new firmware
* write some open-source firmware to fix my bugs and anyone else's

The ICP protocol is badly documented (aka missing), so that needs reversing from the C#/.NET tools; the PCB is missing the necessary pull-up resistors for the serial port (normal anti-reversing / cost saving trick); I don’t know if the ICP allows reading the firmware out (see below!).

Hardware first

The EPF011/021 data sheet indicates serial port pins, which appear to be brought out to solder pads (nice), however the pull-up resistors have been left off and I need to borrow a TTL-serial adapter (thanks Jason!) A bit of soldering later and all is good here. Baud rate - get the ‘scope out and measure the pulse width, looks like ~17usecs/57600 which matches the screenshots in the dodgy manual for the programming software.

Serial traffic

So, does it say anything when powered up? fire up minicom and configure as 57600/8N1. Yes! Hello KL113 software :) I see some debug info at boot, and more when pressing the channel change button. Out of curiosity I poked at my keyboard and was surprised to see a message “DBGCMD Buffer Full”, ah-ha, looks like I can interact in some way with the firmware.. we’ll come back to that!

What about ICP mode (the data sheet says a pin needs tying to Vdd at boot - that too is usefully brought to a pair of pads)? Unfortunately the data sheet says nothing about the device being erased if you enable ICP mode, time to take a risk. Pop a jumper across the pads, and power up: I get a continuous stream of semi-colons on the serial port, probably some kind of synchronisation / detection, and no smoke. Power down, take off the jumper and re-start - it’s back doing normal things - phew!

Take apart the programming tool

So what of the programming software? Well it works on XP or Win7, but sadly doesn’t have the ‘read device’ button enabled. Time to pull it apart too. My tool of choice here is dotNetPeek from JetBrains, a completely awesome decompiler that turns out almost re-compilable C# and a Visual Studio project to work in from the .exe - sweet :)

Now I have a choice, try and fix their code to enable the device read function, or re-implement from understanding the ICP protocol. Given I want to end up with Open Source (not decompiled!) code, I’ll re-implement. Following the rather badly structured application down to the serial port I/O I get the following to read the device:

Bootstrap:
- wait for a sequence of ';' characters (at least 10)
- write '5', wait to see if ';' chars stop. Repeat if not (up to 4096 times)

Read Flash (one 512 byte block):
- set mode (read) and address (multiple of 256)
  - write 139 (dec) followed by (addr >> 8)
  - read byte, confirm 139 (dec)
  - write 85 (dec)
  - read byte, confirm (addr >> 8)
  - write 85 (dec)
- start command execution
  - write 140 (dec)
- read block bytes and ack
  - read 512 bytes
  - write 85 (dec)
- read block end and ack
  - read byte, confirm 136 (dec)
  - write 85 (dec)

..and for the brave to write to it:

Bootstrap (as above)

Write flash (one 512 byte block):
- set mode (write) and address (multiple of 256)
  - write 138 (dec) followed by (addr >> 8)
  - read byte, confirm 138 (dec)
  - write 85 (dec)
  - read byte, confirm (addr >> 8)
  - write 85 (dec)
- start command execution
  - write 140 (dec)
- write block of bytes
  - write 512 bytes
- read block end and ack
  - read byte, confirm 136 (dec)
  - write 85 (dec)

NB: in both cases, the address must be a multiple of 512 (flash block size), or the device will return an error code in place of the command echo.

This seems easy enough to replicate in clean code, and indeed it worked first time, I have a flash dump.

Getting a new toolchain working

I picked the readily available (standard package) sdcc compiler and fiddled with it’s settings to reduce the junk it emits for 8051 targets, writing my own boot.asm minimal startup and a couple of very short test programs (blinkenlight obviously, and a serial port reflector).

The Makefile does a little more fettling with sdobjcopy to convert the Intel hex output of the linker to a binary image, and it works :)

That’s it for now

Code here in Sourcehut.

GEO Trio II In-Home-Display Insides

Sat, 06 Nov 2021 22:14:30 +0000

Updates

2021-11-06: Stupid bug fixed in capture code (doh!), links below are to updated code and corrected log from an IHD connected to the Geo together cloud servers and mobile app - whoo :=) As below, all data exchanged with the cloud is encrypted in some way apart from an obvious timestamp at the start..

2021-08-27: Much better logs and a corresponding packet capture now available, annotated with what I have now worked out - it’s making a lot more sense, but sadly - it looks like the Trio II main device protects all transmitted data before sending to the WiFi module :=(

Annoted log file (prepare for ASCII documentation!)
Packet capture (load into your favourite decoder)
WiFi boot text (might help with hacking firmware?)

Why would you do this?

I have just joined the millions who have 2nd gen (SMETS2) smart metering installed, and was interested in sending metering data to other systems (like home automation), but it turns out the existing standards and devices don’t really support this (which is a bit crap but hey..)

The In-Home-Display (IHD) unit I have is a Green Energy Options (GEO) Trio II, which also works as a Consumer Access Device (CAD), if the optional WiFi module is fitted. Turns out this will only talk to the energy company web site and send them the data for fancy cloud based app stuff - not what I want.

However - if the interface to the WiFi module is easy to understand, then it might be possible to build a special module that can divert the data to a sane protocol such as MQTT. So what’s the module interface?

Here’s what’s in a GEO Trio II

Teardown photos

As you can see, it’s a fairly simple beast, with just a few ICs, the aerial (ZigBee for meter network), the LCD and buttons. The devices are all identifiable (eventually, once you type the right text into search engines).

MCU#1: NXP LPC54605J512ET180 low power MCU / ARM Cortex-M4, 512k Flash, 200k SRAM, USB (x2), many serial/GPIOs
MCU#2: NXP JN5169 wireless micro controller / ARM (unstated), 512k Flash, 32k SRAM, 2.4GHz radio I/O, serial/GPIOs
Flash: GigaDevices GD25Q32CSIG 32Mbit serial flash

The WiFi adapter connects to the 6 pin 0.1" header at the top, two pins (1,2) are Vdd and Gnd, one is likely a module detect line, leaving just three for data - most likely an SPI or UART interface to a full-stack WiFi+TCP/IP package such as this from Radicom: Radicom embedded WiFi Serial Module

..which bodes well for faking stuff, as it may not be encrypted at this point (the TLS happens on the WiFi module) :)

[update, Jan 2021]: it’s not a standard WiFi/serial module, the WiFi board has another user-programmable device, an ESP8285, running more Geo code.. this appears to handle asynchronous comms and has a separate command and control protocol back to the main system. This makes sense if you wanted to use a different network technology later on, and also holds promise that I can still emulate and transmit my own data - quite possibly by reprogramming the existing ESP8285 =) See below…

The unpopulated devices may be related to developing SMETS2 standards for the radio interface, or a change of display technology, moving the LCD controller function to the LCD itself.

Next to the WiFi module connector is another set of pads that are accessible without disassembling the device - these are very likely a JTAG (AVR layout) connector for programming/upgrades, I still need to buzz these out to find grounded pads, which will confirm this guess.

[update, Dec 2020]: the pads are indeed an ARM-10 JTAG/SWD (dual mode) connector, confirmed by grounding of pads 3,5,9 :)

Investigating the WiFi module

[update, Dec 2020]: Courtesy of an email or two with William McNicol (thanks!) I have purchased a WiFi module direct from GEO here: GEO WiFi module plugged it in and confirmed that it’s sending packets to the ’net - rather surprisingly UDP/IP. More info once I’ve got the ‘scope on the connector pins!

[update, Jan 2021]: I finally got round to probing the WiFi module while it’s connected to the main system, and yes, it’s a simple UART @ 115k2 8n1 with both Tx and Rx lines. Attaching an FTDI FT232 at 3v3 logic levels gives me bytes, oh so many bytes… so what to do with them?

[update, Jul 2021]: I forgot to include the pinout for the WiFi module (thanks Tim for the nudge!):

(as numbered on the board itself, looking at the back while connected to the IHD)
3v3 - 2  1 - GND
n/c - 4  3 - n/c
Din - 6  5 - Dout
(where Din = UART Data INto the WiFi board, Dout = UART Data OUT of the WiFi board)

Relationship between serial traffic and network

[update, Aug 2021]: I acquired a second TTL serial to USB adapter, and wrote a packet decoder that gave me much nicer timestamped serial traffic logs (see top of page!), my assumptions below were a little off, but close :=)

historic info

I thought it best to try and understand the relationship between traffic on the serial interface and the network, so I hooked up the FT232 and my network sniffer to a common clock source (thanks NTP!) and watched everything for the first few minutes of IHD operation.

It turns out to be pretty complex - I was able to deduce some of the serial message format, in particular the start message marker, the length and checksum fields, which gave me accurate timestamps for each message, and some content boundaries. Trying to relate these to the UDP traffic on the other hand was not making any sense, it looks like the command protocol simply issues a ‘connect please’ request, or one of a couple of other requests (there are obvious request type codes), and the ESP8285 operates completely asychronously, sending multiple different network packets, none of which appear to contain much from the serial messages. There is some consistent timing that I have noticed:

serial messages are typically at 10 second intervals (per message type)
network packets are at 4 second intervals (per length/type)
network packets are repeated 4 times, each with the same timestamp
network packet timestamps appear to be related to ZigBee / meter time, but I cannot see this transferred in the serial messages when it updates from boot (2011) to now (2020) :=(

Full details are in my log file (prepare for ASCII documentation!)

Libvirt, QEMU & Samba

Thu, 01 Oct 2020 16:05:00 +0000

Trying to be different again?

Well, yes. I have used many virtualisation systems over the years, and many of them ‘just work’, especially well supported tools such as HyperV or VirtualBox, however I wanted to find a solution that was possible in plain Debian stable, no special repositories (eg: Oracle for VBox), no special agents in the guest (again VBox needs this) and fully open source (ruling out VMware and HyperV stuff).

Out-of-the-box virt-manager+libvirt+qemu (also kvm for x86 guests) works quite well for actually running my guest VMs, the networking is well integrated, the GUI (Spice or VNC) is usable too. Choosing qemu as the virtualiser means many target platforms (eg: ARM, x86, Sparc, …) and it tends to be first with new targets too.

The problem I wanted to solve has been around for years, but there hasn’t been a single documented solution: getting a shared folder exposed from my Debian/linux host to a Windows (7) guest VM, while running everything in user-mode, ie: as me. No special privileges or VMs escaping as root!

So what should work?

There are guides, oh so many guides to ramming special software into the guest so it can talk Plan9, which is the only qemu built-in protocol for file system access. This is usually a litany of pain and hideously slow, probably worth avoiding.

What about good old SMB/Samba then? Windows will be fine, just need to install a file server on your host machine and pipe that networking through.. again, many tutorials, that assume you are running stuff as root (default libvirt setup), that your VM will have networking (what if you are investigating malware?) and of course lots of manual messing with smb.conf and trying not to expose your host file system to elsewhere! It’s still slow :(

Interestingly, qemu has built-in pseudo-support for SMB, the -netdev user,smb=/path/to/folder option, as long as you have Samba installed (doesn’t need to be running, qemu does that, plus configuration). This looks promising, but sadly libvirt and virt-manager know nothing about it so I put this aside for a bit..

I found a ‘modern’ alternative, the aptly named Virtio-FS, which promises performance and security, doesn’t need network adapters in the guest and support from RedHat - could be good?

Virtio-FS

I went with modernity, it has to be better than what it’s trying to replace right?

Turns out, it’s a little too modern for Debian stable, I needed a backport build of qemu (v5.0+) to get the necessary support, then of course libvirt/virt-manager know nothing about it, and there isn’t a backport for these - cue much hacking trying to write that supporting logic in bash in a wrapper script around qemu (don’t do this people!)…

Then there’s the assumption in the source for virtiofsd (the actual file i/o daemon) that it will be running as root, because that’s how libvirt operates… nope :( I actually went as far as to pull the source and fix it, but then finally fell foul of a gap in support on the guest - virtio-fs is only supported on Windows 8+ Gah! I did learn a lot about libvirt/virt-manager and qemu however!

Samba (again - but working)

So back to option #2, SMB/Samba, using the qemu built-in forwarding to a local instance of smbd that it configures correctly, all with just one simple command line option, that I now understand how to provide through libvirt/virt-manager - it doesn’t work… Much debugging/rummaging and swearing later I find that libvirt makes assumptions about how you want to run qemu, in particular it enables the security sandbox that stops qemu spawning another program - ie: smbd! With this disabled (undocumented file location BTW), it finally works.

Alleged documentation for libvirt security

What actually needs doing then?

OK, here’s the full list of things to make user-mode SMB work with libvirt/virt-manager/qemu (+kvm):

Install required packages: # aptitude install virt-manager, qemu, qemu-kvm
Create your VM as desired (or import from elsewhere, VBox in my case)
Remove the NIC - you will be putting this back via custom arguments below..
Use the magic virt-xml tool to add custom qemu arguments as per Stack Overflow % virt-xml $(VM) --edit --confirm --qemu-commandline '-netdev' to add a single arg, then % virsh edit $(VM) to get into your chosen text editor and add the following additional lines below -netdev:
```
<qemu:arg value='user,id=hostnet0,smb=/path/to/shared/folder'/>
<qemu:arg value='-device'/>
<qemu:arg value='virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:21:69:da,bus=pci.1,addr=0x0'/>
```
NB: You may want to change my MAC for one of yours :)

Create/edit a user-mode qemu.conf in ~/.config/libvirt/qemu.conf and add this text:

# Override defaults for libvirt when running in a user session
# Turn off the sandbox so we can spawn SMBD..
seccomp_sandbox=0

This should be it - fire up your VM, ensure you have virtio network drivers installed (part of the OS unless you are on Win7 - grab them from Fedora Virtio-Win if you are) and map that drive:

C:\> net use X: \\10.0.2.4\qemu

Whoo hoo!

Apeman H45 Trail Camera Insides

Sat, 04 Jul 2020 17:20:30 +0000

What’s in the box Phil?

I took some pictures

It appears to be the common Sunplus 1628A based reference design shared by many similar Chinese origin trail cameras.

Did you reset the PIN?

Err nope. No sign of a battery, so assuming it writes config to flash, and I was unable to find a button-mashing-based reset process (although there likely is one as the manufacturer offers a telephone reset service)

We got lucky, the owner suggested a few numeric pairs they usually use (I know - poor practice, they know too), and after 20-30 combinations we got the magic value :)

A talk on threat modelling

Sun, 24 May 2020 21:48:36 +0000

Threat modelling is not voodoo

..honest it’s not, but many folks think it’s special and don’t do it :( Thus Phil put a talk together for his local OWASP chapter, which seemed to be well received (you can judge for yourself in the video of course!)

It’s only taken 8 months to get the slides on line!

View raw presentation or click and press 'F' for fullscreen

Here’s me giving the talk (48mins, plus pizza!)

Summerhouse for 2020!

Sun, 19 Apr 2020 13:25:10 +0000

First buy a kit..

Courtesy of the nice folks at Screwfix, and Shire Garden Buildings, we ordered ourselves a new summerhouse, specifically the Aster 12x8 foot model.

We had a delivery date of 29th March, then lockdown happened.. but it turns out deliveries of existing orders are ok, so a pile of parts arrived on our driveway.

Paint everything. Twice.

It’s surprising how much garden space you need to lay out all the parts and paint ’em… also worth noting is that some of these do NOT fit through a standard 78"x30" door frame so we had to carry them over the garage flat roof - you’ll need 4 people for that as the heaviest panel weighs ~80kg.

Prepare the ground and install base

The instructions are really big on having a level base, luckily for us the previous building here took care of much of that, and we were able to re-use the wooden joists it stood on, after flipping them round 90 degrees so they are orthogonal to the timbers making up the floor panels. Also of note - the veranda floor is not connected to the other panels other than at its ends so both it and the main floor panel need support along their join (you can’t see it here but we have a 4th joist doing that bit). In case you were wondering, the fork was left there as a position marker while we flipped the panels over and back to screw them together underneath.

Assemble!

Attach all the walls, starting with the shed structure to give stability, then work along the back across the other end and the front, using a spirit level to check your work as you go and adjust before screwing it all down to the floor. Nice.

Be careful your floor isn’t distorted when making the last join, we had to readjust on discovering the shed section was slightly tilted backwards!

Roof it. Add windows, fix electrics :)

..and we’re done! We took our time, doing a couple of hours a day so we took the best part of a week to finish. It can be done in a day once the parts are painted and dry.

Welcome to the comfy outdoor living experience, yes that’s a BBQ patio to the left, and yes we have power and network installed (I’m working from home in here real soon!)

Downloads

Tue, 21 Jan 2020 21:31:20 +0000

Stufff!

Static sites are cool

Sat, 01 Jun 2019 19:50:13 +0000

You get best of breed cyber security :)

Deployment is always easy (ZIP anyone?)

Content management is nicely orthogonal to deployment.

Version control is whatever you choose - sanity is best.

Blackhole DNS

Thu, 10 Aug 2017 22:16:10 +0000

TL;DR gimme the scripts!

OK, chill… they’re all here: https://git.sr.ht/~phlash/sinkhole

Err why?

Since Christmas Joseph had noticed a number of login attempts to Steam from weird locations, none successful due to using 2 factor authentication (yay!), but concerning as it’s an indicator of compromise. Steam passwords getting stolen / ex-filtrated from somebody, are a cause of concern for the victim as they may have other (more important) stuff going the same way.

Early investigation, by changing the password then carefully sharing it round the family members who have a need to know, did not turn up anything obvious such as leakage occurring directly after a specific individual used it on their device(s). So we thought we would try setting a trap…

DNS for the win

All network traffic, from inside our home to outside, will very likely use a DNS lookup to locate external services before making a connection or sending UDP packets (there are some exceptions of course, more on these later). Since we already operate our own DNS server, for split-brain purposes, we should be able to use this as a filter to pick up any unexpected lookups, record the device that asked and follow up with more detailed investigation. Simples. Almost.

First we need to prevent any DNS requests bypassing our own server, so that’s adding a firewall rule to the home router to block port 53 (TCP+UDP). Turns out this /also/ blocks all ICMP traffic, as an undocumented side-effect, which was rather confusing for a while (thanks TP-Link!)

Next we need to enable logging on dnsmasq (our local DNS service). This done we investigate the logs, and promptly get swamped by the thousands and thousands of DNS requests in a day. We’ll need to add some intelligence!

Blackhole lists

Network intelligence comes in the form of a curated list of ‘bad host names’ from github, courtesy of Steven Black https://github.com/StevenBlack/hosts. Adding this to dnsmasq as a local hosts file prevents lookups going to the real world, instead dnsmasq responds with whatever IP address we choose to put in the file, so we choose 127.0.0.1 (there’s no place like home), and ::1 (IP6 equivalent as some evil scripts do IP6 lookups of CNAME records to bypass DNS blackhole filters like this one).

This turns out to be /really/ effective, no more adverts on devices, much faster load times in many cases, but unfortunately in other cases, no page loading at all. Bah. It seems a number of advert-supported sites (comics mostly) will not load the content until the ad has arrived, or sites rely on responses to their stats trackers (looking at you Azure!). Some exclusions need to be applied to the block list to get everything back up and running.

Automating stuff

Black hole lists change on a daily basis, exclusions need applying and reports need emitting: so I’ve written a couple of scripts to help automate these parts (see github above).

Sinking traffic

Rather than ‘blackhole’ DNS requests with a local IP(6) address, Phil thought it might be fun to operate a true ‘sinkhole’, that is, an endpoint for potentially malevolent TCP/IP traffic, so it can be captured and investigated. Given the nature of the traffic, it is unwise to use a real TCP/IP stack in a live system (there may be exploits arriving!), and it may be useful to avoid the kernel overheads of processing traffic for real (memory mostly), so the fun begins:

Inspired by articles like this https://umbrella.cisco.com/blog/2014/02/28/dns-sinkhole/, and starting with sample code like this https://github.com/jedisct1/iptrap Phil wrote a Python/Scapy based sinkhole program (pysink.py in github repo). This worked, but Scapy is seriously slow and it ground our quad core server to a halt! A second try, this time using the raw socket API and writing all the protocol dissection required in custom code (raw.py in github), works much better. To separate potentially dangerous traffic from normal stuff, we added a USB Ethernet adapter to the server (borrowed from the Wii!) but did not assign an address, thus traffic is not passed up/down to IP or higher layers, allowing the script to fake everything. We /did/ have to disable promiscuous ARP responses (/etc/sysctl.conf: net.ipv4.conf.all.arp_ignore = 1), otherwise the ARP layer would respond to physical broadcast packets on the unaddressed interface.

OK, we’re terminating dodgy traffic to the point where we have the first payload packet arriving, which is usually enough to classify it, so how to do that? Turns out there is an off-the-shelf solution, Suricata! Suricata is a traffic analysis tool/IDS, designed to be fed from the span or tap port of a physical switch, however it will work just as well from the raw interface used to terminate the sinkhole :)

What do we learn/see then? Well, there are several damn noisy bits of kit on the home network (Windows PCs, Humax STB) and Dropbox and Chrome do weird things (broadcasting to detect peers https://blogs.dropbox.com/tech/2015/10/inside-lan-sync/, sending fake DNS requests https://isc.sans.edu/diary/Google+Chrome+and+%28weird%29+DNS+requests/10312). Unfortunately we don’t see any Steam credentials flying about, nor do we detect any known malware (yet).

Catching the problem

So after all that fun (and it was fun!), we are no nearer finding how Steam passwords are leaking out. Joseph made yet another password change late one evening, and didn’t pass it to anyone else as they were all asleep. In the morning we discover it has leaked! This clearly indicates Joe’s laptop as the source, so we dive into the DNS and packet logs to try and find any evidence. After a couple of exciting .cloudfront.net names turn up, only to be discarded as they used by HP (they run cloudfront and dogfood their own CDN), we run out of data in the time period involved. So we must conclude that the malware is carefully avoiding DNS, most likely by operating it’s own peer-to-peer network with bootstrap IP addresses. Much as we’d like to attach Suricata to our outbound router to see all the traffic, it cannot provide a span port (thanks again TP-Link). We also considered putting Joe’s laptop through a real hub (I have kept an old 10M hub for this purpose) and monitoring that for a while, but time was against us, so we take a pragmatic route..

Resolution (by fire!)

Given Joe needed his laptop cleaned up, and we are unlikely to catch the malware in action over a short time period, we took the nuclear option: boot into Linux from a live CD, image the on board drive (SSD) and parts of the game store (HDD), then wipe the partitions and re-install Windows from a known good ISO. While Windows was installing several years worth of patches (at least there is a roll up pack now!), we scanned the images using ClamAV to see if anything turned up. Sadly not, a few false positives discovered by putting hash values through VirusTotal, and that was it :(

Resolution (by Valve)

Trying to be helpful, we contacted Valve support and explained the issue to them, whereupon they asked “which account?”… Turns out Joseph had two, one active and one forgotton - guess which one had a known password in a data breach? NB: Never use the same email address for >1 account!

Bosch SMS Disassembly

Mon, 05 Dec 2016 20:54:30 +0000

Trip!

So last Wednesday the house power tripped out while I was away at just before midnight, Joseph did the right thing, checking for burning things before pushing the trip back on. All appeared well again. Chalk one up to weirdness.

Then on Thursday it happened again, while I was present. We checked for obvious failures again, pushed the trip back on and it popped straight back out this time. Pulling all the plugs around the house got it to stay on, so we worked through the list of ’things we turned on about midnight Weds and 10pm today’ - the dishwasher was the only option. Plug everything except the suspected kit back in - all good, problem identified.

Grab the multi-meter, measure the resistance between live and earth on the plug pins, above 20 Mohm with the power off, below 1 Mohm with the power button in. Time to pull it apart and find the short…

Disassemble!

This is where it gets tricky as there are no disassembly instructions for my Bosch SMS40T42UK/09 that I can find on the ’net and no obvious way in (like screws on the back on the lid). I’ll have to wing it, possibly break stuff and document as I go. If you are going to follow along at home, you’ll need a Torx T20 and a large flat screwdriver. Here’s the teardown photo stream :)

the lid

Two catches, accessible through the rectangular apertures just below the lid hold the whole lid on at the front, which then slides off the back on hooks. I discovered these the hard way by breaking one :(

the side panels

Once the lid was off, the rest was easy, a series of three (different) screws hold each side panel on. All Torx T20 heads. Once the screws are removed, the panels tilt outwards and drop down from hooks underneath.

trouble spotted

No need to remove any more panels, at this point I spotted moisture where it shouldn’t be, above the electronics and in the base tray. What I thought especially odd was that there was a sheet here to deflect water away from the electrics at all, looks like a known issue!

confirmation of short

Pulling the cover off the electrics revealed a wiring loom with one obvious fat connector for the heater, which I disconnected and measured for resistance to the earth / chassis: <1Mohm, confirmation that the heater had lost integrity and making the whole machine suspect.

abandon hope

At this point it looks like a known sealing / leakage issue around the side of the wash tub has resulted in considerable water ingress to the base tray and thus the heater (which practically rests on the tray), destroying its insulation from the outside. Further disassembly and possible replacement of the failed part might resurrect the machine, but its over 3 years old, and the pipework on the other side (not shown) looked pretty manky / furred up, so I abandoned further attempts at fixing and bought a replacement washer. I figured it had depreciated at under 2 pounds/week so didn’t owe me much!

Family

Sun, 05 Jun 2016 00:59:56 +0000

Here we all are…

|| Phil || Stuart || Angela || Martin || Robert || Joseph || Thomas || Helen || Simon ||

..and for those who need to know what we looked like with hair… Phil & Stuart have a collection of Old School Photos for your delectation and delight!

Lenovo Battery Hack

Fri, 03 Jun 2016 19:31:21 +0000

The Itch

Phil bought himself a replacement high-capacity battery for his Lenovo T430 laptop recently. It was from a reputable supplier in Germany, clearly listed the T430 as ‘compatible’ on the product description and was half the price of an official Lenovo battery. What could possibly go wrong…

Upon insertion all looked well, the T430 worked fine, Ubuntu was able to read the capacity and charge level (68%), but after an hour of use while plugged into the AC power it didn’t seem to have charged up (still 68%). Hmmn. Rebooting displayed a warning message:

The battery installed is not supported by this system and will not charge.
Please replace the battery with the correct Lenovo battery for this system.
Press the ESC key to continue.

WTF!

I contacted the seller who promptly offered a return of the item, and a postage refund once I had a receipt for posting. So I repacked the battery and popped down to the post office, who told me that Royal Mail won’t handle a battery pack without it’s associated ‘device’ aka my laptop. Great, so back on the ’net to find a delivery company who will ship a battery to Germany. No one. Nowt. Zilch. At least, not for less than the cost of the thing (£40)!

Research

Searching the ’net for the displayed error message it becomes clear that Lenovo laptops have a ‘Smart Battery’, and that from the T430 onwards, they implemented a challenge/response authentication mechanism to lock users into buying Lenovo supplied parts, ostensibly to protect systems from battery fires when ‘fast charging’ the pack. Seems like 3rd party manufacturers are unable to replicate this behaviour properly, and some (thanks INTENSILO) mis-label their batteries as compatible when they are not.

It also turns up the splendid work of zmatt http://www.zmatt.net/unlocking-my-lenovo-laptop-part-1/, who had the exact same problem on his X230, but chose to reverse engineer the problem rather than return the battery - good man!

A little later I come across more good work by Hamish Coleman https://github.com/hamishcoleman/thinkpad-ec, who has automated the earlier work of zmatt and added other patches to support different keyboards on multiple IBM/Lenovo laptops, including the original battery fix for the X230.

The Scratch

Since I have no sane way to return my 3rd party battery, and prior work exists which is /almost there/ to fix my problem I decide to go for it (what’s the worst that can happen? I’ll brick my laptop):

Step 1: Replace the original battery and check all is well (apart from the rubbish capacity).

Step 2: Carefully work through zmatt’s BIOS unpacking and editing descriptions, double checking with Hamish’s Makefile and tooling, and teaching myself to drive radare2 (also awesome), I arrive at a point of no return: I /think/ I have a correctly patched BIOS file, there is only one way to test it: flash my laptop and hope for the best… I’m good: it starts up :)

Step 3: Swap to the 3rd party battery and see if it charges.. yep! Relief!

The Payback

It’s how open source works: I’ve got a new capability in front of me - one that can fix battery charging issues on T430 laptops with a specific BIOS version (2.57 if you’re interested). I have to do the decent thing and work that into Hamish’s tools to give back to the community.

[edit: pull request accepted] Go get it from Hamish’s Github above!

Rivendell_Chroot

Mon, 28 Dec 2015 10:20:27 +0000

New holiday - new self-imposed challenge!

Context

My favourite radio station FXR use Rivendell Audio as part of their play out system, and have recently been donated an [HP TouchSmart 300-1025 PC](http://h20564.www2.hp.com/hpsc/doc/public/display?docId=c01918320 “wikilink”) to use in the public area, which I was asked to 'make work' for audio management duties and general Internet stuff..

The Patient

Turns out that HP ship the !TouchSmart system with Windows 7 Home Premium, loaded with lots of their own sloooow software (thanks to a not-exactly-rapid hard drive and heavy use of multiple .NET versions), then the owner / donor adds another boatload of cruft like Apple iTunes / Bonjour, ezRecover, several dodgy browser plugins (the usual suspects

adware, search bars, etc.), several browsers (hello Chrome + Google services) AVG and Norton. The result is a machine that takes 10+ mins to become usable from a cold start and frequently stalls as background tasks hog the I/O bandwidth... not nice.

Fix Attempt 1: Remove the crud, switch to Security Essentials for anti-malware, prune the browsers back to Firefox only and see if it's usable.

Fix Result 1: I bricked it (almost) by uninstalling Bonjour, which seems to completely hose the IP stack. After the uninstall, Windows event log service fails to start and /lots/ of other stuff depends on the event log - broken O/S :( System restore got things back working, then I successfully cleaned out a lot of other junk with help from !SysInternals tools (autoruns is fabulous!), and Process Hacker. Unfortunately it wasn't much more usable in this state, and the Windows version of Rivendell software is unable to play audio, which doesn't make good use of the hardware capabilities. Rethink time.

Fix Attempt 2: Ubuntu 14.04 LTS live CD, re-size NTFS partition(s) and install basic GUI system, then add Rivendell packages. Simples.

Fix Result 2: Ubuntu just worked, almost. Everything except the touch screen was supported out of the box, including the !WiFi card and camera, and as expected it was pretty quick, even after choosing the new Ubuntu GNOME 3 desktop. Some additional rummaging on the 'net produced a working touch screen too :)

Obsolete app

Now I hit a problem - the Rivendell Audio version we use @ FXR is officially obsolete (1.7.2 circa 2010) and I have lost the original packages, not that they would install on 14.04 as the dependencies are all wrong..

I can of course query the existing installation(s) for file lists, and collect all the binaries & libraries together. I can also grab the dependency lists, but how to avoid collisions when I need an old library to be used and I have a newer one in 14.04?

Collision Avoidance

My first thought was simply to drop the app and it's libraries (down to libc) in a folder, hack the LD_LIBRARY_PATH and see if stuff worked... kind of, except the app is 32bit, whereas my host O/S was 64bit, and many libraries depended on supporting config files that collided with the host. Hmmn.

Some Googling later, it became apparent that what I needed was a chroot environment, something I tend to avoid as my previous experience with such things has been fragility and continuous maintenance work (I prefer to avoid that!). However, there are now tools available to make stuff easier, in particular the fabulous debootstrap which can build a working chroot version of any Debian or Ubuntu distribution, and schroot which manages all the fragile bind mounts and authentication file copying automatically as you enter/leave the chroot - nice :)

The Chroot

With a bit of assistance I built a chroot environment targetted at 32bit Lucid Lynx (10.04) to match the FXR studio systems, I chose to install the minimum system, which includes apt-get so I can add other components easily. I then worked through the dependencies from the app, and installed appropriate packages to satisfy these. So far so good, but I had no package for the app itself..

The Hack

Having satisfied the dependencies I then copied the remaining binaries into place from the archive taken from the studio - did it work? Yep :)

Unfortunately Rivendell Audio relies on a number of daemon processes, and other GUI tools to operate, whereas a default chroot expects only transient programs, so some additional work was required, in particular creating a session via schroot to host the daemons, then re-attaching to that session for each transient program. Much learning curve!

Pulse Audio

Yes the daemon from hell :) However it's mostly useful for normal programs, I just need to get a daemon running in a chroot to use it via ALSA bindings.. sounds easy huh? Well it turns out that other folks have had trouble before, the only reliable approach is to enable network access and use it - other approaches that rely on DBus and shared memory seem to be fragile. So enable network access, limit to localhost and we're done :)

Session on Demand

Last bit - when to run the schroot session and daemons? Given that we depend on pulseaudio and that in turn runs per user login, I chose to have startup scripts for each transient program check for a session and create it.. then on logout it gets torn down. Job done.

FUNcube_2nd_Birthday

Sun, 22 Nov 2015 14:55:02 +0000

In celebration of FUNcube-1 (http://funcube.org.uk/) having been operational for 2 years, Helen made a cube cake, which we deployed :)

"It's a cube cake!"

That critical moment as the antenna are deployed..

Success - contact established :)

We just need to make a slight adjustment to the software...

Happy 2nd Birthday FUNcube-1!

OBiLINE

Sun, 07 Jun 2015 21:00:25 +0000

1. page was renamed from ObiLINE

OBiLINE USB FXO Adapter

Those lovely folks at OBIHAI have recently produced the cheapest USB FXO device I've seen: OBiLINE, available from Amazon all over the world, but thus far nobody seems to have taken the lid off - until now :)

||<tablestyle="float:right;" style="padding: 1em;":>||

What's in the case?

Not a lot it turns out (kind of expected since it's £25.00 inc. VAT!).

Parts

NuvoTon Nano120LD, ARM-Coretex M0 Micro + USB Datasheet
SiLabs 32178-FM1, Telephone interface chipset Datasheet
SiLabs 32919-A-FS, paired with above to provide FXO parts

What does it look like on USB

Here's the interesting part: the device works in conjunction with OBIHAI's closed-source VoIP gateway products (e.g. Obi202). One guy plugged one into his Windows PC http://www.obitalk.com/forum/index.php?topic=7236.0, but didn't get much info.

I've just bunged mine into my Ubuntu lappy, and it turns up as a sound card (S16_LE, 8kHz fixed), no other control interfaces are apparent, so I don't know how it's used to seize the line (perhaps by just opening the device?) or how ring detect works. I will attach the other end to a PSTN line next and see what happens!

Can I make it work?

Short answer: no :(

I've tried playing with declared audio controls via alsamixer - nothing much happens. Opening the device for record or playback results in a stalled device.. there is something that needs to happen to wake this thing up.

lsusb dump:

root@lenny:~# lsusb -vs 3:5

Bus 003 Device 005: ID 4f42:6901  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x4f42 
  idProduct          0x6901 
  bcdDevice            1.00
  iManufacturer           1 Obihai
  iProduct                2 USB Audio Device
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          192
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    [[MaxPower]]               64mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      1 Control Device
      bInterfaceProtocol      0 
      iInterface              0 
      [[AudioControl]] Interface Descriptor:
        bLength                10
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdADC               1.00
        wTotalLength           70
        bInCollection           2
        baInterfaceNr( 0)       1
        baInterfaceNr( 1)       2
      [[AudioControl]] Interface Descriptor:
        bLength                12
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bNrChannels             1
        wChannelConfig     0x0001
          Left Front (L)
        iChannelNames           0 
        iTerminal               0 
      [[AudioControl]] Interface Descriptor:
        bLength                 8
        bDescriptorType        36
        bDescriptorSubtype      6 (FEATURE_UNIT)
        bUnitID                 5
        bSourceID               4
        bControlSize            1
        bmaControls( 0)      0x03
          Mute Control
          Volume Control
        iFeature                0 
      [[AudioControl]] Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             2
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               5
        iTerminal               0 
      [[AudioControl]] Interface Descriptor:
        bLength                10
        bDescriptorType        36
        bDescriptorSubtype      6 (FEATURE_UNIT)
        bUnitID                 6
        bSourceID               1
        bControlSize            1
        bmaControls( 0)      0x01
          Mute Control
        bmaControls( 1)      0x02
          Volume Control
        bmaControls( 2)      0x02
          Volume Control
        iFeature                0 
      [[AudioControl]] Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0301 Speaker
        bAssocTerminal          0
        bSourceID               6
        iTerminal               0 
      [[AudioControl]] Interface Descriptor:
        bLength                12
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             4
        wTerminalType      0x0201 Microphone
        bAssocTerminal          0
        bNrChannels             1
        wChannelConfig     0x0001
          Left Front (L)
        iChannelNames           0 
        iTerminal               0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
      [[AudioStreaming]] Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      1 (AS_GENERAL)
        bTerminalLink           2
        bDelay                  1 frames
        wFormatTag              1 PCM
      [[AudioStreaming]] Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      2 (FORMAT_TYPE)
        bFormatType             1 (FORMAT_TYPE_I)
        bNrChannels             1
        bSubframeSize           2
        bBitResolution         16
        bSamFreqType            1 Discrete
        tSamFreq[ 0]         8000
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0100  1x 256 bytes
        bInterval               1
        bRefresh                0
        bSynchAddress           0
        [[AudioControl]] Endpoint Descriptor:
          bLength                 7
          bDescriptorType        37
          bDescriptorSubtype      1 (EP_GENERAL)
          bmAttributes         0x00
          bLockDelayUnits         0 Undefined
          wLockDelay              0 Undefined
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
      [[AudioStreaming]] Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      1 (AS_GENERAL)
        bTerminalLink           1
        bDelay                  1 frames
        wFormatTag              1 PCM
      [[AudioStreaming]] Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      2 (FORMAT_TYPE)
        bFormatType             1 (FORMAT_TYPE_I)
        bNrChannels             1
        bSubframeSize           2
        bBitResolution         16
        bSamFreqType            1 Discrete
        tSamFreq[ 0]         8000
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes           13
          Transfer Type            Isochronous
          Synch Type               Synchronous
          Usage Type               Data
        wMaxPacketSize     0x0100  1x 256 bytes
        bInterval               1
        bRefresh                0
        bSynchAddress           0
        [[AudioControl]] Endpoint Descriptor:
          bLength                 7
          bDescriptorType        37
          bDescriptorSubtype      1 (EP_GENERAL)
          bmAttributes         0x80
            [[MaxPacketsOnly]]
          bLockDelayUnits         0 Undefined
          wLockDelay              0 Undefined
Device Status:     0x0000
  (Bus Powered)
root@lenny:~#

Hi-res photos

ScopeHacking

Fri, 18 Apr 2014 18:50:26 +0000

Have DSO, Will Hack :)

I've got a lot of stuff in the house (that I should probably chuck, but hey), including a Netgem iPlayer and it's associated infra-red full qwerty keyboard. The iPlayer unit is pretty much useless now, but the keyboard might be fun as a remote for an under-TV PC, I just need to know the protocol to program LIRC..

So how to decode an IR protocol? Well, I've got a Digital Storage Oscilloscope, and found a photo transistor in the box of bits - hooking them up with a couple of resistors for bias and a 5v supply (USB cable) gets me an IR receiver:

This happened to be pointing at my dimmable lights when connected, which showed me a nice 100Hz hum :-)

A bit of fiddling with the controls and I can now detect the signal coming from the keyboard, a pulse train of IR carrier bursts at 38kHz (the usual), with variable width pulses in blocks repeated at intervals while I hold a key down.

Part 1: Pulse timing

I adjusted the 'scope such that one pulse train block fits into the screen width, set the trigger point at the left edge of the screen and selected single capture mode, then hit 'Space' and grabbed the pulse train into storage (all timing in milliseconds):

|| on || 2.4 || || 0.8 || || 1.6 || || 0.8 || || 3.2 || || 2.4 || || off|| || 0.8 || || 0.8 || || 0.8 || || 0.8 || || 2.4 || ||

Hmmn. Looks like everything is a multiple of 0.8 msecs - so let's convert those timings to bits and see if it looks like a serial protocol:

|| Space || 111010110101111000111 || 21 bits? Not sure, let's try another key || || Menu || 11101011010100100111 || 20 bits? Less sure now, try another one.. || || News || 11101011010111001111 || 20 bits again. First 11/12 are the same.. ||

If this was using partiy and start/stop bits (sensible given the unreliable IR carrier), there should be 11 bits/byte, which fits in with the first 11/12 bits being the same (address byte followed by keycode perhaps?). Let's divide the bits up, include a final stop bit (0) and check if a parity bit works:

|| Space || 1(S) 11010110(V) 1(P) 0(T) / 1(S) 11100011(V) 1(P) 0(T) || Looks like even parity.. || || Menu || 1(S) 11010110(V) 1(P) 0(T) / 1(S) 00100111(V) 0(P) 0(T) || Checks out, still even.. || || News || 1(S) 11010110(V) 1(P) 0(T) / 1(S) 11001111(V) 0(P) 0(T) || Could be right here :) || || LShift|| 1(S) 11010110(V) 1(P) 0(T) / 1(S) 01110011(V) 1(P) 0(T) || Yep - rekon I'm right! ||

So it looks to be 1200 baud serial data with even parity (aka 8E1), two bytes per pulse train block.

Part 2: Block repeats & press / release bit

Are blocks repeated to provide resilience? Or just while a key is held down?

Re-adjusting the 'scope, I can see that holding a key down produces a pulse train block every 100 msecs, by capturing two blocks and zooming in (gotta love DSOs!), I can see that the same pulse train is emitted while a key is held, this gives me a nice key-repeat feature if I want one.

What about if I just tap a key?

I can just about hit and release a key in <100 msecs, the resulting blocks have a short spacing between the first two blocks, then 100 msecs to the third block, then nothing. Zooming in again shows that the second and third blocks are the same, but are different from the first, like this (omitting start/stop/parity bits):

|| Menu 1st Block || 11010110 / 00100111 || || Menu 2nd Block || 11010100 / 00100111 ||

Ah-ha! We've found the press / release bit in the first byte, and we also know that release blocks are always sent twice.

Protocol - hacked.

StackGrabbing

Fri, 21 Feb 2014 08:45:09 +0000

Grabbing the Stack for Fun & Profit

So I'm at a work, trying to track down some memory leaks in a long-running VC++ application (actually a COM object DLL out the back of a .NET web service - you don't need to know that bit). No problem, I'll use the Microsoft C Run-Time library memory debugging features.. a quick read / web search later, I have the necessary incantation in a common code header:

 #define _CRTDBG_MAP_ALLOC
 #include <stdlib.h>
 #include <crtdbg.h>
 #define new new(_NORMAL_BLOCK,__FILE__,__LINE__)

..all goes well, the application terminates and I get a dump file of leaked memory blocks: 34MB of text, with very few file and line numbers. Of those entries that do provide file and line numbers, they're all just a few places, buffer management classes that could be called from anywhere. Back to square one.

OK, so I need more context, in particular I need to see the stack when those allocations are made. No problem, I can use a debug memory allocation hook function, take a stack back trace for each alloc, remove it for each free then dump what's left at the end.

Attempt 1: StackWalker

So - how to grab a stack back trace? I first came across some excellent but rather involved code from Jochen Kalmbach: http://stackwalker.codeplex.com/ which he wrote primarily for printing a crash dump stack trace with lots of extra info.

This turns out to be pretty easy to use, but rather slow, mostly because it does symbol lookups while walking the stack. I'm seeing ~20 million allocations during a test run, and !StackWalker is tracing at about 100/sec, so a test run would take 200k secs or about 2.3 days. Try again.

Attempt 2: CaptureStackBackTrace

My Google fu improved after attempt 1, and I found CaptureStackBackTrace, a kernel/RTL function that does exactly what it says on the tin - fabulous and fast, no symbol lookups or callbacks to read process memory, using this made no discernible difference to memory allocation speed - winner!

CRT Oops

So now I can run a test - result: ~20 million allocations, followed by zero frees. Nothing. Not one. That can't be right can it?

Back to the debugger to trace what's going on in the memory allocation hook function where I'm supposed to be matching allocation requests with free's and removing those from the list to print out. Turns out that when Microsoft wrote the debug memory library they chose to not provide the hook function with any useful information when callers free memory blocks, so I'm unable to associate free's with allocs. Genius. A little more digging and some bodgery results in me being able to extract the allocation request ID and record memory leakage with stack traces like this:

#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
#include <DbgHelp.h>

// stolen from dbgint.h - because M$ in their wisdom don't provide matching request serial numbers between alloc/free hook calls. Sheesh.
#define nNoMansLandSize 4
typedef struct _CrtMemBlockHeader
{
        struct _CrtMemBlockHeader * pBlockHeaderNext;
        struct _CrtMemBlockHeader * pBlockHeaderPrev;
        char *                      szFileName;
        int                         nLine;
#ifdef _WIN64
        /* These items are reversed on Win64 to eliminate gaps in the struct
         * and ensure that sizeof(struct)%16 == 0, so 16-byte alignment is
         * maintained in the debug heap.
         */
        int                         nBlockUse;
        size_t                      nDataSize;
#else  /* _WIN64 */
        size_t                      nDataSize;
        int                         nBlockUse;
#endif  /* _WIN64 */
        long                        lRequest;
        unsigned char               gap[nNoMansLandSize];
        /* followed by:
         *  unsigned char           data[nDataSize];
         *  unsigned char           anotherGap[nNoMansLandSize];
         */
} _CrtMemBlockHeader;
#define pbData(pblock) ((unsigned char *)((_CrtMemBlockHeader *)pblock + 1))
#define pHdr(pbData) (((_CrtMemBlockHeader *)pbData)-1)

// require dbghelp.lib for SymXX() calls
#pragma comment(lib, "dbghelp.lib")

...

// collect stack traces during allocation, remove when free'd
static int maxreq=20000000;
static ULONG skipframes=5;
static ULONG maxframes=10;
typedef struct {
    PVOID trc[10];
    ULONG len;
    ULONG cnt;
} stack_t;
static stack_t stacks[20000000];

static volatile BOOL inWalk = FALSE;
static long acount = 0;
int __cdecl allochook(int type, void *usr, size_t size, int block, long req, const unsigned char *file, int line) {
    wchar_t msg[80];
    if (inWalk) // ignore recursive requests
        return TRUE;
    inWalk = TRUE;
    acount++;
    if (acount%10000==0) {
        wsprintf(msg, L"%ld\n", acount);
        OutputDebugString(msg);
    }
    if (req<maxreq) {
        DWORD hash;
        switch (type) {
        case _HOOK_ALLOC:
            // insert into map - alert collisions
            if (0==stacks[req].len)
                stacks[req].cnt=1;
            else
                stacks[req].cnt++;
            stacks[req].len=CaptureStackBackTrace(skipframes, maxframes, stacks[req].trc, &hash);
            if (stacks[req].len>maxframes) {
                DebugBreak();
            }
            break;
        case _HOOK_REALLOC:
            // update/insert into map - may collide
            stacks[req].len=CaptureStackBackTrace(skipframes, maxframes, stacks[req].trc, &hash);
            if (stacks[req].len>maxframes) {
                DebugBreak();
            }
            break;
        case _HOOK_FREE:
            // AWOOGA! AWOOGA!! HIDEOUS BODGE ALERT!!! Recover request ID from CRT debug memory header because
            // Microsloth DO NOT PROVIDE IT IN THE HOOK FUNCTION CALL. Numpties.
            if (usr==NULL) {
                OutputDebugString(L"NULL [[UserData]] in free hook\n");
                DebugBreak();
            }
            req = pHdr(usr)->lRequest;
            // remove from map - alert missing entries
            if (0!=stacks[req].len) {
                stacks[req].len=0;
            } else {
                wsprintf(msg, L"free without alloc ID: %d\n", req);
                OutputDebugString(msg);
            }
            break;
        }
    } else {
        OutputDebugString(L"Oops: out of request array!");
        DebugBreak();
    }
    inWalk = FALSE;
    return TRUE;
}
static void dumpstacks(_HFILE hFile) {
    char buf[4096];
    DWORD nw;
    int cnt=0;
    HANDLE hProc = [[GetCurrentProcess]]();
    wcstombs(buf, path, wcslen(path));
    SymInitialize(hProc, buf, TRUE);
    wsprintfA(buf, "------ stacks\r\n");
    WriteFile(hFile, buf, strlen(buf), &nw, NULL);
    for (int req=0; req<maxreq; req++) {
        if (0!=stacks[req].len) {
            int o;
            wsprintfA(buf, "{%d/%d}: ", req, stacks[req].cnt);
            o=strlen(buf);
            for (USHORT stk=0; stk<stacks[req].len && o<4080; stk++) {
                char tmp[256];
                char tmp2[sizeof(SYMBOL_INFO)+256] = {0};
                SYMBOL_INFO *pSym = (SYMBOL_INFO *)tmp2;
                pSym->MaxNameLen = 255;
                pSym->SizeOfStruct = sizeof(SYMBOL_INFO);
                if (SymFromAddr(hProc, (DWORD64)stacks[req].trc[stk], 0, pSym))
                    wsprintfA(tmp, "%s (%08x),", pSym->Name, pSym->Address);
                else
                    wsprintfA(tmp, "%08x,", stacks[req].trc[stk]);
                strcpy(buf+o, tmp);
                o+=strlen(tmp);
            }
            WriteFile(hFile, buf, strlen(buf), &nw, NULL);
            WriteFile(hFile, "\r\n", 2, &nw, NULL);
            ++cnt;
        }
        if (req%1000==0) {
            wsprintfA(buf, "%d/%d\n", cnt, req);
            OutputDebugStringA(buf);
        }
    }
}

...

int main() {
...
    memset(&stacks, 0, maxreq*sizeof(stack_t));
    _CRT_ALLOC_HOOK oldHook = _CrtSetAllocHook(allochook);
...
    hFile = [[CreateFile]](L"leaks.txt", GENERIC_WRITE, FILE_SHARE_READ, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    _CrtSetReportMode(_CRT_WARN, _CRTDBG_MODE_FILE);
    _CrtSetReportMode(_CRT_ERROR, _CRTDBG_MODE_FILE);
    _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE);
    _CrtSetReportFile(_CRT_WARN, hFile);
    _CrtSetReportFile(_CRT_ERROR, hFile);
    _CrtSetReportFile(_CRT_ASSERT, hFile);
    OutputDebugString(L"Dumping stacks\n");
    dumpstacks(hFile);
    OutputDebugString(L"Dumping memory leaks\n");
    _CrtDumpMemoryLeaks();
    CloseHandle(hFile);
    return 0;
}

..fun eh? But this at least works (whilst eating ~1GB of RAM for the tracing buffer!).

Leakage Found!

After all that entertainment, I finally had ~180k memory blocks that weren't free'd on program termination, this time with stack traces, all with a common ancestor method - victory is mine! I won't bore you with the details of the bug (besides it's proprietary code from work), but I will say that ATL COM objects are ugly voodoo, and rolling your own makes stuff much clearer (and not dependant on the paid for version of Visual Studio).

Projects

Sat, 21 Sep 2013 22:25:15 +0000

Illuminated Wesley Guitar

[

] (/articles/illuminatedguitar/)

Bob decided to spend his 16th birthday money on a new axe, but not just any axe - oh no - a clear acrylic Wesley Strat copy (PE200wh). Of course a perfectly clear acrylic block just begs to be lit up from the inside. Oh yes it does :)

Pond

[

] (/articles/thepond/)

In an effort to move our pets outside, to encourage more wildlife in the garden and to grow more interesting plants, Angela, Phil and the family have built a pond. Bring on the herons!

Shed

[

] (/articles/theshed/)

Constructed between Oct/Dec 2005 to replace the mouldy old shed in Phil and Angela’s back garden, “the shed” is intended as a recording studio / drum practice room and general bolt hole :) Getting the panels into the garden was entertaining…

[

] (https://ashbysoft.com/maps/Roberts_Map_Page.html)

Bob has been busying himself with the latest Valve Software Hammer editor, producing several new maps for the Counter-Strike:Condition Zero game..

[

] (/articles/theconservatory/)

Built between April and November 2003 (but not finished until 2005..), this was Phil’s first serious building project, and it was quite a challenge!

Slash

Mon, 10 Jun 2013 23:51:29 +0000

All About Stuart

Hi ! I’m the guy on the right. If I look like my brother that’s because we’re Twins. If you haven’t worked it our already, I’m an IT geek into badminton, music, coding, games, hacking, fast cars and occasionally skiing and sailing. Strictly on holiday only of course..

In police terminology I’m 1m70, 80Kg, hazel eyes, short/receeding hair, glasses. Definite axe-murderer material. [That's 5ft7 and 12st for all you imperialists out there.] I prefer ‘cute’ myself.. Just don’t call me ‘middle age spreading’ or I’ll have to kill you.

Here’s what I’m like:

I don’t smoke (unless I’ve already finished too much Jack Daniels),
I don’t take recreational pharmaceuticals (unless everyone else is)
I used to play badminton. I want to start again…
I drink occasionally (i.e. when anyone buys me a pint). Usually right after badminton!
I enjoy adventure, experiencing the power of nature, the local culture, becoming one with the world; in particular with the local home-brew.
I play guitar; when I can be bothered to practice.
I ski and sail when on holiday. I should practice both to get better but I don’t
I’m quite funny. No really ! This text was written too late at night..

Here’s the low down. It get’s better. Honest!

Play

Serious playing requires a little time away from the office, and a social life. I try to make time for that, but I’ve got this lot to contend with first:

Love Life Amazing! I am now married to a loverly Cornish girl called Gayle; I suspect she might be a Piskie so I will have to watch my step :)
Family Even more amazing! I have a baby son called James, born September 2011, and baby girl called Megan born Jan 2013. So proud of Gayle. Big hug!
Music I am a Practicing musician; well perhaps ‘practice’ is too strong a word - perhaps ‘fools around’ is more accurate! Most recent outing as the bass player in the works Xmas band I USED to I play guitar in an ’electric folk’ kind of combo; think Pentangle but more Irish. You can find out more about pH5 I also try to write/record stuff and I have a little project studio in my back room. Digital of course :)
DIY I make stuff; furniture mostly, in a rather modern, modular kinda style. Think Tapley, if you know that brand, but with more handles..

I also try to keep my house sorted, and the car, and the PC network, … current projects are to renovate the garden into a zen-like low maintainance area, and re-build the digital with linux instead of Uncle Bill’s nightmare O/S

Helen

Fri, 29 Mar 2013 17:48:36 +0000

Email: «helen@ashbysoft.com»

PhilsDroid

Tue, 12 Mar 2013 11:45:54 +0000

New Droid for Old

So I lent my droid phone (Huawei U8120 below) to Joseph, 'cause his shiny toy got pinched :-(

Upon mentioning that I was now droidless, one of my colleagues at work (thanks Duncan!) has lent me his unwanted Samsung GT-I5500, of a similar vintage to the U8120, but locked to Three.. uh-huh.

Unlocking the GT-I5500

According to the interweb (well ok, mostly the excellent XDA Developers Forum), the network unlock code in a GT-I5500 is stored in the ROM image in plain text - nice.

All one has to do is temporarily root the device using rageagainstthecage or similar, connect to an adb shell as root, dump the ROM from /dev/bml5 using cat or dd, then pull it off the phone and search for a well-known signature near the unlock code. Power cycle the phone and type the code in when prompted. Done.

The Stock ROM

It's running Froyo (2.2), with Samsung bells and whistles and of course the G-man spyware. I'll be wiping this and putting ICS on when I get a chance. There are also a bunch of Three apps (games mostly) that need to 'authenticate' via 3G (ie: prove you are still using their network!) - not played any of course since I am on Vodaphone (BT Mobile).

Old Droid

A colleague at work was kind enough to offer me a free, reasonably modern handset (Vodafone 845 / Huawei U8120), so I'm now the owner of an Android phone - how hip is that :)

This page is here to chart my progress in updating/modifying/hacking/developing etc. If I learn anything that wasn't obvious from the 'net I'll let you know.

In The Beginning

It was running the stock Vodafone build of Eclair (Android 2.1), the bootloader was not locked (dunno if Huawei bootloader can be locked), the recovery OS was stock Huawei and it worked. Time to break it...

NB: Getting to the recovery OS is a case of holding the 'Call' or green button while powering it on. Getting to the bootloader requires holding the 'Hangup' or red button while powering it on.

Backups and Ice-Cream

Some rummaging on http://forum.xda-developers.com tells me that I can run Ice-Cream Sandwich (ICS/Android 4.0.4), via an unofficial beta release of CyanogenMod9. Of course I'd like to keep the stock OS incase I ever need to put it back.

The thread on XDA developers was very clear on proceedure:

start in bootloader mode, connect to a machine with the bootloader control application (fastboot), then replace the recovery OS with !ClockWorkMod.
put CM9 installation package on an SD card
restart in recovery mode, with the SD card installed
take a NANDroid backup of remaining flash partitions
install CM9 from package on SD card

Take deep breath and restart in normal mode - wait a looong time while Android configures. It works!

Interesting Diversion -> Joe's Droid :)

Joseph has just got himself a shiny new HTC Desire C on contract (so now I'm waay less hip), and wanted to backup and root it (as you do). A little more research turned up our Rooting method of choice.

Applying this method gained us a root shell (via adb), from which we were able to copy off the raw flash partition of the recovery OS using dd from the appropriate /dev/block/mmcpXXX device to /sdcard/recovery.img. That accomplished we installed !ClockWorkMod and took a full NANDroid backup. Nice job.

UI Pointers

Ok, so I've got to get used to the UI metaphors on this touch screen device (never had one before). Here's a list of all the things that I didn't find obvious:

Unpairing bluetooth devices: Get to settings->bluetooth, then touch and hold the icon next to the device name. All the instructions I found on line forgot to mention the icon behaves differently to the device name area. Update: this appears to be a CM9 anomaly only - but I would still have expected someone to mention it (CM is popular!)
Tethering to provide Internet access for another device: settings->more->tethering, enable. However, do not then go and play with the connection options (accessed via the icon as above), as this reverses the tethering and the phone expects the device to provide it with 'net access. Took me a long while to work out what was going on there!

Network Debugging

New for ICS, it supports network connections to the debugging daemon (adbd), so once I finally got it tethered to my laptop (see above), I could connect to the phone's IP address and get a wireless shell. Yes this also works over WiFi, but you wouldn't want to expose an unauthenticated root shell to the 'net would you? eh?

Photos...Err where?

I took a couple of photos to check the camera works ok (it does), but couldn't find them anywhere on the phone with the File Manager app.. odd. Turns out I had assumed the DCIM folder on my SD card was a hangover from previous use, and that the Pictures folder was the obvious place. Nope. Android follows the JEITA specification and puts photos in DCIM. And videos. And the Gallery app in CM9 beta is broken so I couldn't look at photos on the phone. Now using QuickPic which seems to work ok.

Robert

Sat, 10 Nov 2012 16:33:28 +0000

Robert smells. Also he makes musics https://soundcloud.com/a66448579

Joseph

Sun, 08 Apr 2012 22:46:55 +0000

Email: Me

This is Joe and I’m going to make a proper page whenever I can be bothered and have found something to write about :)

Hacking Picasa3

Tue, 13 Mar 2012 01:00:25 +0000

The Need

I have a whole pile of photos of family and friends that I have indexed with Picasa v3.8 and it's face recognition ability, using the Windows version of Picasa3 (because it was the 'right' way to do this a couple of years ago).

In the middle of some facebook'ery using my preferred OS (Ubuntu 10.10 ATM) I decided I wanted a photo of a friend to post, so grabbed the Picasa installer and ran it up in wine. All went well, but I got a nice empty list of albums and no people, despite telling it where the photos are on the network file server. Time to find out where Picasa hides the metadata :)

The Research

Of course I went straight to the 'net and found a couple of posts back in 2007 that talked about copying album information from the Application Data area of the Windows user to the same area in wine's emulation: kudos to these guys:

http://forensicir.blogspot.com/2007/07/picasa.html

http://smallutilitiesforfree.blogspot.com/2007/11/how-to-copy-picasa-albums-from-windows.html

The Hacking :)

A little prodding around on my Windows partition shows that pretty much the same structure is still there in current Picasa3 builds, along with the all important Picasa2/contacts folder that has a nice XML file with all my people in, and their Picasa hash values. These hash values are what ties together the contact details and the faces in photos, which are stored in special files (.picasa.ini), one for each folder through the image archive itself (on the file server).

I therefore copied the contacts.xml file, and the album XML files (from Picasa2Albums/) from my Windows disk to wine's emulation area (~/.wine/...).

The next trick is to convince Picasa3 that I have the same mapped drive letter (S:) in wine for the file server that I have in Windows - easy to arrange with winecfg.

The final hack was to open up the Picasa2Albums/watchedfolders.txt file and remove the slightly broken assumption that I have C:\Documents and Settings within wine..

The Result

Startup Picasa3 again, and lo! I have albums with photos in, but still no people. Bah. Just out of curiosity I tried adding a folder and Picasa did it's thing, scanning for faces and then oh joy reconnecting the faces with the contact details. So the fix is simply to rescan all the photos and it will pick up all the existing data (slowly). Time for bed while it chews through 10000 photos!

Phil.

pH5

Sat, 01 Oct 2011 14:39:35 +0000

Slightly Acidic

What do you call a sharp-edged, slight funky/quirky electric folk/rock band (anything, they can’t hear you - ed.)? Ours was called pH5 (geddit?), and it existed from ~1994 - 1999. Band members:

Dr. Simon Phoenix: keyboard wizard, technohead, muso and lyricist.
Stuart: guitars, percussion, engineering and vocals
Phil: bass, percussion, engineering.
Maurice Patrick, violin - just that, but bloody good at it :)

New(ish) Material (2009!)

Back in 2009 - Phil uncovered a couple of un-mastered tracks recorded by pH5 back in late ‘99 or thereabouts - probably in Stuart’s house, just before the band separated to pursue their own interests, so for your listening delight, I mixed them for release here:

Old Material

For your entertainment, here are the tracks from our one and only half-decent studio album, recorded in 1996 at the Driveway Studio in Trimley St Mary (cheers Colin!):

Driveway Demo 10.96.m3u The playlist for…
Musical Priest.mp3
Mad Moll Set.mp3
Pirate’s Waltz Drowsy Maggie.mp3
Daily Bikini.mp3
Bay Of Funday.mp3
Caislean an’ Oir.mp3

CD’s available on request for a minimal reproduction fee of £2.00 - please contact Phil if you would like one.

CMI-8738 vs Jack

Sun, 17 Apr 2011 18:39:34 +0000

The Problem

A few years ago I bought an impressively cheap CMI-8738 based sound card from Trust, it cost £15 I think. Ever since I have been trying to find a way to use it reliably under Ubuntu linux for digital recording from our Akai DPS-12.

The root cause is that as soon as you enable the S/PDIF input, the card stops delivering sample buffers via Alsa unless there is a valid S/PDIF signal present (ie: the DPS-12 is connected and switched on). This is fine except that we are using jackd at the bottom end of the audio stack, and jackd doesn't like a card that doesn't sample regularly, it crashes out with a watchdog error.

We could remember to switch the DPS-12 on before the PC every time we switch the PC on... not gonna happen unless someone intends to record something, or we can live with no digital input (as we have been doing for a number of years).

The Solution

..for which I owe Mr Torben Hoen a beer: alsa_in

It's so darn simple - leave jackd running with output only (S/PDIF and analogue are fine and nice low-latency), then in my audio startup script, run alsa_in to read samples from the S/PDIF input and pretend to be the standard capture backend for jackd (ie: call it 'system').

The magic is that alsa_in is happy to work with no input, it just delivers empty buffers to jackd unless the S/PDIF signal is provided, thus samples arrive from the CMI-8738, whereupon alsa_in delivers those to jackd - genius :)

/me dances a little jig :)

ZTE-G X930 Insides

Sat, 12 Feb 2011 17:18:44 +0000

I took Tom's phone apart

Since there do not appear to be any internal photos of the ZTE-G X930 mobile on the 'net at the moment (and it's not FCC registered) and Tom broke off his USB socket, requiring the phone to be dismantled, I took the opportunity to photograph the insides, enjoy (click for larger version):

Not very surprising really - it's the same !MediaTek ARM-based processor as many other Chinese mobiles (MT6235BA), and thus likely runs the Nucleus Plus OS like the Sciphone i68+. The soldered in battery is a bit unusual - probably internal RAM backup (in which case we'd better hope Tom had nothing irreplaceable in there!)

RivendellHacking

Mon, 31 Jan 2011 19:01:07 +0000

Rivendell Hacking

Open source software is great, even when it has faults, since one can fix it oneself and return the patch to the community right? Well almost, here's what I've been up to with the play-out software used at the local community radio station http://www.felixstoweradio.co.uk:

Win32 Binary Fixes

Short story

It seems none of the Rivendell developers are interested or able to build Win32 binaries at the moment, so I had to patch the binary program using IDA Pro to prevent a crash when creating new logs.

Get the patched file here: attachment:rdlogedit2.exe

Long story

About a year ago we installed the Windows binaries to allow our editing / production teams to create play lists from the comfort of their desktop(s) in the station office, or indeed at home. Unfortunately we found that it crashes when attempting to create a new play list (aka a 'log'): so I grabbed the source, a helpful stack dump from wine and settled down to fix the fault. The problem wasn't hard to find, a simple null pointer dereference because the Win32 version doesn't support multiple users, but someone forgot and referenced the currently logged in user name variable. All I had to do now was build a fresh binary and test it - this is where it all came unstuck as for some reason the developers had chosen to use a closed source version of QT (3.21) on Win32, so I had no means to rebuild the binary. Never mind, I'll just post the patch to the developers mailing list and ask them to rebuild it. I was duly thanked for finding the problem and lo! a new version of the source code appeared with the fix - but no Windows binaries :(

I waited for 6 months, pestered the developers slightly and waited some more - still no new binaries. It seems whoever had the right build tools and libraries (ie: a QT3.21 license) has gone away. So now I have to choose between trying to port everything to QT4 (which is open source on Win32), as suggested may be possible on the Wiki, or do something ugly

patch the binary :)

Well I chose the later since it was going to be quicker and we need it working now: a brief sojourn with IDA Pro, and the excellent advice from Marco Ramilli did the trick, overwriting the pointer reference with nop instructions and inserting a reference to the constant user name instead. Surprisingly this just worked. The fixed file is attached above.

Simon

Sat, 04 Dec 2010 23:22:49 +0000

SIMON

Simon Profile:

Age: 8
Height: About 4ft
Weight: Quite heavy
Hair colour: Blonde
Eye colour: Brown

Shoot on sight, he is armed and highly dangerous.

Reward for capture: £500

PlayingWithFire

Fri, 08 Oct 2010 08:24:13 +0000

Playing With Fire Official Website. This page is no longer needed :)

FoodPyramid

Sat, 27 Jun 2009 21:50:31 +0000

This appeared on our shopping list blackboard at some point last week:

Heh :)

TheShed__PartIII

Tue, 14 Apr 2009 11:34:37 +0000

Interior construction

So the plan (which I will at some point scan in and put here!) goes like this:

Install hidden wiring for lights and power.
Line the walls and ceiling with good quaility acoustic / thermal insulation and plasterboard.
Install lights and heating!!
Build a partition wall across the middle, at an angle, with a double door and window to create an isolation booth / drum room.
Insulate the booth floor with jabloc or similar underfloor polystyrene slabs.
Pave the booth floor to provide an acoustic mass, line with ply or MDF.
Install acoustic absorbers and diffractors in the booth to control resonances.

Simple huh? Well yes, but it’s a slow and fiddly job and frequent stops for tea to warm our hands up don’t really help :)

Install Wiring

This was actually rather easy - once I had decided where the lights needed to be, it was just a case of tacking in the cables, marking the ends so I could tell which is which once the lining is in (thus hiding the cables), and getting hold of some 3-core + earth for the switch (which ended up coming from a wholesaler on a 50m roll, of which I used 2m :)

Line the walls / ceiling

After some research (ie: surfing the ’net for hours) I chose to use Crown Acoustic Partition Roll in conjunction with ½" plasterboard which should achieve 20-30dB of sound insulation and excellent thermal insulation too.

That dangerous looking chap on the right is Martin practising holding up a bank with a staple gun… silly boy.

Installing said lining is taking a while, but we’re half way there now (11th Jan ‘06). The ceiling (see above) gave us the most ’entertainment’ since a whole sheet of ½" plasterboard weighs about 40kg; Martin and I tried holding that over our heads for several minutes while Joseph rushed up and down the ladder putting the screws in! Not good. It’s much easier with four people holding the board up (thanks to Rob and Angela :)

Here’s Stuart putting up the final piece of lining on the front wall, it was starting to get quite warm inside by this point.

Onward to the results!

Big40

Tue, 14 Apr 2009 11:34:28 +0000

Stuart’s Party

Hedonism! Well sort of, you’ve gotta slow down a bit at 40… Thus Stuart and several of his delightful friends had a rather grown up party, and invited Angela & Phil along too. Apparantly we brought the camera, I don’t really remember much in detail!

The lovely lady on the sofa to the left is Charlotte, who was eventually persuaded to bring her parents along so she had someone to sit on. Being of the fairer sex, and slightly underage for most of the drinking games, Lottie retied to bed around 10:30, having stolen the show until that point.

On the right Stuart waxes lyrical about the issue of indigestion in modern society, and provides some fine examples of his subject :)

If I remember correctly (although things were getting a tad hazy at this point), this is Phil, Marie and Stuart trying to work out “what IS that thing on the table?” It must have been good, Marie is taking a photo (hmmn - must ask her for that one!) I strongly suspect we ate it shortly afterwards…

Eeek! It’s a madman with a bunch of forks! Runaway! Actually I suspect it’s just Stuart offering cake to all and sundry - after all it soaks up Jack Daniels nicely - hic

Posh Nosh

Having sobered up from the earlier adventures, Phil & Stuart thought it was about time to treat the family to dinner at a nice restaurant. Thus we all headed over to St Albans and the Sopwell House Hotel for dinner in their delightful abode.

We’ll start at the end then shall we? This is the birthday boys themselves, either side of a coffee-revived Tom, for some reason Angela suggested we take our specs off for this one.

Yes it’s Stuart talking again! I rekon he was telling a joke at this point, since Dad is laughing and Mum has that slighty quizzical look that says “is this a clean joke?” Hehe. You can tell from the mince pies and plentiful empty glasses that we are still at the end of the meal.

It’s Anne, all sparkly and surrounded by those very large glasses again (can’t think why). Martin is modelling the latest in ‘can I borrow a tie’ fashion, and the expression of a seasoned rock star when approached by yet more paparazzi - More photos? Oh all right then…

It’s the cheeky boy himself! Doing his best not to look like a school physics teacher (yes he really was one of those), Jim is wearing a vintage pinstripe shirt, and that special tie that says “Mr Higgins - I’ve lost me pencil Sir!”

Ohhhh! Is that a camer-flash! Yup. Bob has decided that shaving is for wimps, so the tash is likely to get longer… when it reaches the same density as his eyebrows he might notice! Joe is demonstrating the supreme calm that can be achieved by drinking the finest champagne and stealing Grandad’s mince pies.

So errr when do we get fed exactly then Mum? Poor Helen has been waiting all day for decent meal, and we made her sit in the car for 3 hours before she could eat. It’s a hard life sometimes. I promise she was much happier when the food arrived!

ThePond

Tue, 14 Apr 2009 11:34:28 +0000

Ah summer - a gentle breeze wafting through the trees, the trickle of water in the stream, the splash of frogs frolicking in a pond… ah. Better dig one then

It’s October, so the nasturtiums are running riot :) Can you see the fish?

<– the North end

the South end –>

Pretty isn’t it?

2 out of 3 of the fish we put in survived, and then, rather satisfyingly, we got little fish, and lots of ’em :) We’ll tell you how it goes over the winter…

TheShed__PartII

Tue, 14 Apr 2009 11:34:28 +0000

Demolition

Removing the old shed took ages, mostly because we had to carefully take off the roof panels, while still standing on them, or the very wobbly supporting rafters, then strip out all the rockwool lining (boy does that stuff make you itch!) before we could properly enjoy the task of hacking down the walls with a crowbar and big hammers :) In the end we filled 2 skips (of 4 yards each), and had some left over to burn - ahhh the smell of woodsmoke in the evening!

Construction

Day 1

After the ritual burning, we cleared the area, relocated and levelled the paving slabs, then laid new joists on them, finally I aligned the joists carefully before we placed the floor panels on them.

Now the fun bit - putting up wall panels that are 10 feet long, 8 or 9 feet high, and weigh between 50-70kg each, in a force 4-5 breeze. Suffice to say that we nearly lost the whole lot over next door at one point - Robert managed to stop it from leaving the garden, I then grabbed a nearby lump of 4x2 which I used as a lever to put the structure back on it’s base and Martin fitted a couple of screws into the floor which helped a lot- phew!

Having secured the walls to each other and the floor, we fitted the central roof rafter and propped the structure using some left over timbers from the old shed, then retired for the evening - worn out.

Day 2

Pleased to see the structure still where we left it, we tackled the task of putting the roof panels on, from inside the building with one step ladder, one exterior ladder and lots of cunning.

Luckily the panels fitted nicely through the gap between the walls and the central rafter, and we were able to push them up, then slide them back into position. Joseph then got to work with the drill and fitted the securing coach screws before the wind picked up again.

Our final construction task of the day was to put down the roofing felt which Thomas and Joseph did very nicely with just a bit of guidance from Dad

Angela then set to work with the woodstain, turning our building a lovely shade of green. The result? See for yourself!

..and now..

Detailed design is now starting on the internal room, which is probably going to involve a diagonal partition wall (to reduce reflections and gain a little extra space), paving slabs and polystyrene underfloor insulation, ventilation ducts in (floor level) and out (ceiling), more rockwool, acoustic curtains and plasterboarding.. to say nothing of the timber frame to support all this gubbins. I think I’ll enjoy this bit!

Interior construction follows… more »

TheShed__PartIV

Tue, 14 Apr 2009 11:34:28 +0000

The Results!

Well here it is - the (almost) finished product - a recording studio in a shed!

Control room

Here is the control room, from the outside doorway looking in. Nice curtain huh? That’s to absorb any front-to-back reflections at mid / high frequency. Bass dispersal is achieved by putting shelves on the back wall and filling them with junk and spare polystyrene blocks from the floor :)

The ventilation grille serves two purposes - ventilation (obviously), and it’s our wiring conduit between the control room and the booth, which minimises the number of holes through the partition wall.

The control room window is a double glazed conservatory roof vent, mounted sideways. It was left over from building the conservatory. Sometimes I’m glad I horde old stuff.

Yes the speakers really are hung up on bits of string…

Sound booth

Here is the sound booth from the back corner looking towards the control room window. I ought to be honest and point out the not-quite-finished ceiling, which should also be sound insulated but isn’t (editors note: it is now, Jan 2008). As you can see, the main insulation material at the moment is carpet underlay, hung on the interior wooden frame that forms the booth.

The interior control room window was saved from the old shed. Just underneath it is a small shelf holding a cheap MP3-capable mini system for our headphone monitoring amp, and below that is where the wiring comes into the room.

The mixer on the chair and one of a pair of powered PA speakers provide local foldback for rehearsals. The mixer also supplies two line level signals from the microphones / guitars into the control room.

The studio in use

It’s time to rock this joint! Here Martin is demonstrating there is indeed room to swing some sticks in the sound booth. You can now see the other interior window (also reused from the old shed) on the North facing side of the building and letting in a reasonable amount of natural light, but no direct sunlight.

The floor (just visible here) consists of a set of plywood boards, taped together with the indispensable carpet tape (aka Gaffa tape).

Yup it’s Martin again, this time looking cool in front of the partition wall door which leads into the control room.

Of course Bob is super cool when playing guitar too… :)

As indeed are many of their friends, who seem to turn up quite often for organised and random musical events. As long as the food holds out we’ll be fine…

The interior door is just a cheapo from B&Q, with more glass to let in the light from the control room, via the control room partition door, which is double glazed (yet another bit of the old shed recycled).

Here you can see the edge of the 100mm Jabloc underfloor insulation for the sound booth.

Oh, just incase you were wondering, the door is upside down!

Welcome to Shed#1, step this way please…

Thomas

Tue, 14 Apr 2009 11:34:28 +0000

Rock’s :)

hello all people, Tom here, my band is going better than I expected, we’re called

highly corrosive ! we rock (or I think so) :) B) {OK}

I will be back for more stuff later enjoy a bowl of:-

does not compute this web page will self destruct in {3} {2} {1} booooooooooooooooommm!!!!! {X} {X} error rebooting page please wait 50 hours {X} {X}

IlluminatedGuitar

Tue, 14 Apr 2009 11:34:27 +0000

1. intro

Bob decided to spend his 16th birthday money on a new guitar, and he chose a rather unusual instrument, the Wesley Guitars PE200wh, a clear acrylic strat copy - cool innit?

Of course we couldn’t just leave it at that could we? Oh no. You see it’s such a lovely clear piece of acrylic, it just had to be illuminated from the inside. No really, it did…

If you’re gonna put lights inside a guitar, you gotta see what’s already there :)

Here we are opening it up to check the size and shape of the cavity, which is nicely routed, and all the hardware is attached to the frontplate (so no major disassembly required).

Some rummaging on the Maplin website turned up the ideal light source - apparantly designed for lighting around vehicle bodies, these flexible LED strips are low power, super bright, can be cut into smaller units and will bend round inside the cavity.

https://www.maplin.co.uk/Search.aspx?criteria=N56CF&DOY=13m12

To provide a bit of colour, I bought two strips, one white and one blue, and a toggle switch to select between them.

After much careful measuring, some cutting and a fair bit of soldering on wires, the light strips can be taped into the cavity like this. Note that I have soldered the strips with the negative sides commoned together, which cuts down on wires and prevents nasty short circuits if the parallel strips touch.

In order to select different colours, we put in a SPDT toggle switch with a centre detent (ie: off). To this we connected the feeds to the lights, and a link wire to the power input, which comes via…

The necessary 12V supply was fed to the guitar via the middle ring of this stereo jack socket, a stereo screened cable and a custom made breakout box on the floor, to which a suitable 12V supply is connected (we’ve been using an old PSU from a force feedback steering wheel).

Although this worked, it wasn’t ideal for a couple of reasons: the pickup signal and lights share the ground connection at the socket, so ground lift due to lighting currents causes clicks in the signal; pulling the plug in/out momentarily short circuits all the contacts on the plug which could damage both the supply and the amp depending on what shorts first. Not good.

We subsequently replaced the Strat-style socket mounting plate with a flat acrylic plate, and added a standard 2.5mm power socket. We also changed to using a switched mode power supply from an old LCD monitor which neatly avoids any mains hum. It’s now quiet when operating and safe from short circuits - still using the stereo socket with the intention of sending digital lighting control signals to the guitar…

Need I say anything? Just how cool is that? :)

Seeing just how darn bright this is, we are now planning on putting a dual dimmer circuit in the guitar, with perhaps an audio sensor chip (National Semi LM4970 or the like) so the lights follow your playing and/or using the middle ring of the stereo socket to deliver a UART signal from a remote lights controller (DMX via a PIC to the I2C bus input of LM4970 maybe?). Your suggestions are welcome!

Martin

Tue, 14 Apr 2009 11:34:27 +0000

Email: «martin@ashbysoft.com»

Hi, I am Martin, member of the Ashby family, physics student at Warwick University, and wannabe musician. I will reply to email I find interesting, and I also have facebook.

Life:

Consists of getting along with everyone if possible, particularly my girlfriend Georgia. At some point in the near future I will realise that guitar hero is not the path to passing my degree and do some work. In the meantime I proceed with normal student/teenager like activities, plus walking, cycling and minus some of the drinking.

Friends:

I do have some, somewhere :) they know who they are and most of them have facebook or some means of communication available to them. Also I probably owe some of them money…

Future:

In no way do I attempt to predict the future, but after my degree I will inevitably need to pay off student debt. I hope to do this by moving in with Georgia plus friends and getting some kind of real work. I would much prefer to do this by starting some kind of musical career, writing and/or performing interesting and varied bits of popular music entertainment. As a back up option I have always considered robbing a bank or pulling off some major fraud, although now this is publicly disclosed I would hasten to add that it is very unlikely.

Music:

I describe myself as wannabe musician purely because I am not a pro. I have however been playing drums for nearly 8 years, guitar for 5 years and other instruments as and when people have shown me how to play them. I have featured in numerous musical escapades, most recently playing bass guitar and drums in Quorn. Since joining Warwick Uni I have become an active member of the drumming society, and have developed a taste for samba.

Education:

I attended St Albans RC High school in Ipswich, including 6th form, and came out with some A levels and GCSEs: A’s in maths, more maths and physics, and a C in French. I am now on a 3 year course in Physics at Warwick University as previously mentioned, which is pretty interesting most of the time, and not too much work so far. This will change no doubt.

Entertainment:

For entertainment I have many options, most preferable is adventuring with Georgia. Recent times have seen us walk around 20 miles or something like that, see some amazing sights and find tyre swings. Jamming with Robert is another favourite activity, and others to follow include guitar hero in Whitefields 6, drumming sociey sessions, cycling and other sports, and just random things that happen really.

NB: If you are Carl Free, please get a life.

TheConservatory

Tue, 14 Apr 2009 11:34:27 +0000

The Team

Here’s a list of all the lovely folks who helped with this project:

Len (Phil’s dad) - planning, labouring, site foreman - you know the score :).
Jim (Angela’s dad) - moral support, digging & tool hire (pick axe anyone?).
Stuart - more labouring, moral support.
Martin, Robert & Joseph - enthusiasm and construction.
Angela - design, tea making & painting.
Clem - yet more labouring, gophering..

A big thank you to everyone, if you feel I have forgotten you please email me and it shall be corrected!

The Theory

“We need more space!”

Ring any bells? Well with our size family we need all the space we can get, especially when the youngest leaves our room. Right now moving home is simply out of the question - a five bedroom property costs 3-4 times as much as our present 4-bed home did 5 years ago - madness! We had been thinking about a conservatory on and off for a couple of years and now it made sense as the cheapest way to get more space for the ‘stuff’ that accumulates around children (and Phil), freeing up bedroom space for beds! In addition, Angela wants to grow more exotic plants and fruits (bananas), Phil is keen to make the house more ’eco-friendly’, using solar energy from a conservatory for spring/autumn heating - so we went for it!

Start by digging a hole..

This photo was taken at the end of April, two weekends into the project, and two weeks before the kit of parts arrived from our chosen conservatory company Baltic Pine

Actually Phil should have started digging about a month before, because there was a lot of rubble in there that needed removing to the skip (two skips eventually..)

Now fill it in again..

Having dug a nice hole, fill it in with freshly mixed concrete. To a neatly marked out level - or failing that until you run out of ballast.. :)

Buy some more ballast, then some more cement. Then spend another weekend shovelling it into a mixer..

NB: Always hire a cement mixer - even for ‘just one’ bag of ballast..

Build a wall..

..or get a nice chap down the road to build one for you, after you’ve been to several builders merchants trying to find one that has the required type of bricks (a condition of the planning permission of course!)

Cheers Ron!

Relax :)

OK. So at this point we took a holiday - things stopped for a couple of weeks while we enjoyed the delights of Kent.

Paint some woodwork..

Time to unpack the kit of parts that arrived in mid-May, and see just how many bits there are (eek!). Then sort out those that will be going up first and set to work with the wood stain. Nice.

See if the frames fit..

A scary bit. Is the wall the right size? Is everything square? Where does this bit go? Have I sawn off the right end? Phew.

Build the floor

Yup - more concrete! Lovely. Only this time we’ve added a damp proof membrane, underfloor polystyrene insulation, steel reinforcing mesh and it’s all got to be done inside the walls, then levelled off with a batten. We chose to fix some guides to the walls first (a good idea - thanks Dad!).

It was tricky, we ran out of ballast yet again and it was hot, so the cement dried out very quickly - however hosepipes are great! We finished off with a plain cement screed (a thin layer on top of the concrete) to make everything smooth.

By this time the builders merchant / hire firm knew what I wanted as I walked in the door - actually they were getting a little worried about my project and asked if everything was ok, which was nice of them!

Get some help, then some more.

At this point lots of people are needed to stain the woodwork, drill the wall fittings, trim the end frames, apply the sealant, balance the frames, drill the pilot holes, put in the screws, read the manual (again!) until you get this:

Put up the roof

The most complicated bit of any wood conservatory is the roof - in our case a beautifully made pine frame (I am on commision if you buy one!) with double glazed panels, pre-built in the factory, and reassembled on site by yours truely, which must be put back together exactly right or the glass won’t fit - ulp!

Glazing the roof

Today we put in some of the glass panels - and thankfully they fitted nicely!

Update: Unfortunately the aluminium opening roof window was the wrong size - bah. Spent 2 months chasing the supplier for the right size one.

The finished article!

It’s done (nearly)! All the windows are in, the doors are fitted, and the electrics are installed. Which of course means I can have lots of flashing Christmas lights strung up round the roof to show it off at night :) It does however need the dripping box gutters sorting out - serve me right for putting a join along the bottom of the lining :(

This picture was taken at 08:34 on Dec 3rd 2003 - chilly isn’t it?

Would I do it again?

Weeeeeelll… maybe. If so, I would try and design the footings such that the concrete could be delivered ready mixed (however this means ordering at least 1 cubic metre of the stuff at a time), and I’d open all the packages as they are delivered and check the sizes of everything! I’d also line any box gutters with both the bitumen-like flashing strip I used here, and a butyl liner to ensure it doesn’t leak.

Snow update! (2004)

I’m glad to report that after the recent snow falls here in Feb 2004, that my roof construction holds up, and the gutters didn’t leak after I sealed them with liquid rubber back in November. Angela has also been busy painting the walls, and filling it with plants - infact it almost looks habitable, now where did I put that floor tile catalogue….

..and finally.. a floor! (2005)

It’s nearly June 2005, so the structure has been up almost two years, and all that time I’ve been meaning to tile the floor. No really, I have. Honest.

Well I finally discovered a long weekend, at the start of half-term when we weren’t rushing off to do stuff as a family - so I laid the floor - yay!

PS: That slightly sticky up tile in the left foreground? It got fixed last week, and I grouted it all, with more help from Stuart - ta.