Index:


SysAdmin Tips and Tricks

  • Creating a new cert using my CA

    I run my own personal CA, so that I can issue certificates for internal and management services, such as Dell iDRAC web interfaces, web connected terminal servers, internal Grafana and Prometheus, etc.

    Originally I was using OpenSSL’s CA functionality, but I recently moved to using Smallstep CA as it is much friendlier.

    It is not meant to be a comprehensive guide, but rather a quick reference for myself and so I can copy and paste.

    Creating a certificate

    To create a certificate, you can use the `step` CLI tool. The basic command is:

    $ step ca certificate --offline --san localhost --san example.com 192.0.2.1 test.example.com.crt test.example.com.key
    

    Inspecting a certificate

    To inspect a certificate, you can use the `step certificate inspect` CLI tool.

    The basic command:

    $ step certificate inspect --short test.example.com.crt                                                   
    X.509v3 TLS Certificate (ECDSA P-256) [Serial: 2898...3200]
      Subject:     127.0.0.1
      Issuer:      Kumari Intermediate CA
      Provisioner: warren@example.com [ID: Odw5...Aisg]
      Valid from:  2026-03-08T08:11:53Z
              to:  2036-03-05T08:12:53
    

    List issued certificates

    To list issued certificates, you can use the `step ca list` CLI tool. The basic command:

    step-badger  x509Certs ~/.step/db  
    Serial number                            Subject             Start                 Finish                Validity
    302993289828619663782218509723082042948  CN=text.kumari.net  2026-03-08T04:05:50Z  2026-03-09T04:06:50Z  Valid
    288747816944534935195190911414687768787  CN=text.kumari.net  2026-03-08T04:06:41Z  2026-03-09T04:07:41Z  Valid
    256594984883023829367820058718959011064  CN=text.kumari.net  2026-03-08T04:13:24Z  2026-03-09T04:14:24Z  Valid
    110820592971243850069853866489777680598  CN=text.kumari.net  2026-03-08T04:18:56Z  2026-03-09T04:19:56Z  Valid
    290518310107845670357853935107535723354  CN=text.kumari.net  2026-03-08T04:19:50Z  2026-03-09T04:20:50Z  Valid
    185857471328644270079771979276526259328  CN=text.kumari.net  2026-03-08T04:24:13Z  2036-03-05T04:25:13Z  Valid
    289868830197733569134057562717999143200  CN=127.0.0.1        2026-03-08T08:11:53Z  2036-03-05T08:12:53Z  Valid
    210934104339491430029165189931100840639  CN=127.0.0.1        2026-03-08T08:41:06Z  2036-03-05T08:42:06Z  Valid
    28982137696241536685507136660775104074   CN=127.0.0.1        2026-03-08T08:43:38Z  2036-03-05T08:44:38Z  Valid
    
  • Removing an IP address from fail2ban

    I use fail2ban to stop annoying things like ssh brute-force attacks, etc. This isn’t really to stop the attack, but rather to stop the attack traffic from filling up my logs.

    Sometimes I accidentally manage to get my own address blocked, or need to unblock a specific address.

    Step 1: Confirm the fail2ban really is what is blocking the address:

    user> sudo fail2ban-client status sshd
    Status for the jail: sshd
    |- Filter
    | |- Currently failed: 0
    | |- Total failed: 3
    | `- File list: /var/log/auth.log
    `- Actions
    |- Currently banned: 1
    |- Total banned: 1
    `- Banned IP list: 192.0.2.1
    

    To remove the IP:

    user> sudo fail2ban-client set sshd unbanip 192.0.2.1
    
  • Removing bullets from slides.com / reveal.js

    I often use reveal.js (or slides.com) to made presentations. 

    I like to use bullets (unordered lists) to organize information, but often don’t want the circles. This can be accomplished using the CSS editor to style the bullets with the following:

    ul {
    list-style-type: none;
    }

    In order to only apply this to certain elements/text blocks, simply create a new target and set the Class Name to ‘nobullets’:

    .nobullets ul {
    list-style-type: none;
    }

     

     

  • Backing up a whole machine using Rsync, without root-login

    For easy cut-n-paste:

    rsync –stats -av –exclude={“/dev/*”,”/proc/*”,”/sys/*”,”/tmp/*”,”/run/*”,”/mnt/*”,”/media/*”,”/lost+found”,”*/.cache/*”}  -e “ssh” –rsync-path=”sudo rsync” wkumari@vimes.kumari.net:/ vimes_rsync

     

    The cute trick here is to call rsync as ‘sudo rync’.

     

  • ZFS tips and tricks

    Quick notes on creating a ZFS filesystem.

    root@ron:~/src/configs/gitosis-admin# zfs create tank/data/scratch
    root@ron:~/src/configs/gitosis-admin# zfs set quota=10G tank/data/scratch
    root@ron:~/src/configs/gitosis-admin# touch /tank/data/scratch/.duplicity-ignore
    root@ron:~/src/configs/gitosis-admin#

    Creating .duplicity-ignore will prevent Duply (well, Duplicity) from backing up this directory.

     

    Checking the compression ratio:

    zfs get all  | grep compressratio
  • Adding a user to a group (e.g sudo)

    For some reason I have a hard time remembering the arguments to usermod to add a user to a group. 

    sudo usermod -aG sudo wkumari

  • Making DANE / TLSA records.

     Quick reminder for myself on how to generate / update TLSA records.

    ~/local/src/swede/swede/swede create --output rfc --usage 1 -s 0 -m 1 www.kumari.net
    No certificate specified on the commandline, attempting to retrieve it from the server www.kumari.net.
    Attempting to get certificate from 198.186.192.250
    M2Crypto does not support SNI: services using virtual-hosting will show the wrong certificate!
    Got a certificate with Subject: /serialNumber=l/YjABq5T5eemHk7J4kqJviHIR11OOkx/OU=GT03082892/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.kumari.net
    _443._tcp.www.kumari.net. IN TLSA 1 0 1 8d930a464843e08660e3fd1ddce8ed4269cc0cd9cd53a8a306bce8abcf47aef5
    

     

    For the IETF one (tied to a CA)

    ~/local/src/swede/swede/swede create --output rfc --usage 0 -s 0 -m 1 -c ~/tmp/certs/starfield.crt www.ietf.org
  • Creating / renewing SSL Cert

    Generate the new key

    root@vimes:/etc/ssl# openssl genrsa -aes256 -out ./private/star.kumari.net-20110723.key 2048
     Generating RSA private key, 2048 bit long modulus
     ......................................................................................................+++
     .................................................................................................+++
     e is 65537 (0x10001)
     Enter pass phrase for ./private/star.kumari.net-20110723.key:
     Verifying - Enter pass phrase for ./private/star.kumari.net-20110723.key:
    

    Get the info from the old CSR

     root@vimes:/etc/ssl# openssl req -in star.kumari.net.csr -noout -text
     Certificate Request:
       Data:
           Version: 0 (0x0)
           Subject: C=US, ST=Virginia, L=Sterling, O=Warren Kumari, OU=Warren Kumari, CN=*.kumari.net/emailAddress=warrenkumari.net
           Subject Public Key Info:
               Public Key Algorithm: rsaEncryption
               RSA Public Key: (1024 bit)
                   Modulus (1024 bit):
                       00:a1:a1:f4:8a:50:e3:71:ee:4e:d2:3d:51:97:2c:
                       [SNIP]
    94:e9:1f:e7:07:e1:90:1e:ab Exponent: 65537 (0x10001) Attributes: a0:00 Signature Algorithm: sha1WithRSAEncryption 71:b5:82:16:4f:7e:c9:f8:e9:3e:55:fe:86:d9:b9:e9:13:a2: [SNIP]
    23:7d:1f:68:38:5d:ca:12:f9:1e:44:3c:e4:47:a5:be:09:ac: 0b:6b

    Now generate a new CSR

     root@vimes:/etc/ssl# openssl req -new -key ./private/star.kumari.net-20110723.key -out star.kumari.net-20110723.csr
     Enter pass phrase for ./private/star.kumari.net-20110723.key:
     You are about to be asked to enter information that will be incorporated
     into your certificate request.
     What you are about to enter is what is called a Distinguished Name or a DN.
     There are quite a few fields but you can leave some blank
     For some fields there will be a default value,
     If you enter '.', the field will be left blank.
     -----
     Country Name (2 letter code) [US]:
     State or Province Name (full name) [Virginia]:
     Locality Name (eg, city) [Sterling]:
     Organization Name (eg, company) [Warren Kumari]:
     Organizational Unit Name (eg, section) [Warren Kumari]:
     Common Name (eg, YOUR name) [Warren Kumari]:*.kumari.net
     Email Address [warren.at.kumari.net]:
     Please enter the following 'extra' attributes
     to be sent with your certificate request
     A challenge password []:
     An optional company name []:
    
     root@vimes:/etc/ssl# ls
     certs  openssl.cnf  private  star.kumari.net-20110723.csr  star.kumari.net.csr  wildcard.kumari.net.csr  www.kumari.net_godaddy.csr
    

    And provide the CSR to the CA

     root@vimes:/etc/ssl# more star.kumari.net-20110723.csr 
     -----BEGIN CERTIFICATE REQUEST-----
     MIIC4jCCAcoCAQAwgZwxCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhWaXJnaW5pYTER
     MA8GA1UEBxMIU3RlcmxpbmcxFjAUBgNVBAoTDVdhcnJlbiBLdW1hcmkxFjAUBgNV
     [SNIP]
     e+WZXl16+MwNDk0tBQsOn2Z0ppC60O42wouMOIMJD904WS/72/NbDsxVmkmfig/Y
     UqrjcdnOXYfkzOfitv2TWlMwW7WtGQ==
     -----END CERTIFICATE REQUEST-----
    

     

    Installing / using the new certificate

    If you end up with a certificate that needs an intermediate certificate, there are a few options.

    Apache

    Apache knows how to deal with these using the SSLCACertificateFile option.

           # We want SSL for this site.
           SSLEngine On
           # Cert and key locations
           SSLCertificateFile /etc/ssl/certs/star.kumari.net-20110723.crt
           # Intermediate cert.
           SSLCACertificateFile /etc/ssl/certs/RapidSSL_CA_bundle.pem
           # And the key...
           SSLCertificateKeyFile /etc/ssl/private/star.kumari.net-20110723.key
    

    Postfix (and others)

    Some software doesn’t understand handing out intermediate CA certificates, but this can sometimes be worked around by putting both the certificate and the intermediate cert in one file.

    Create the combined pem file:

      $ cat star.kumari.net-20110723.pem RapidSSL_CA_bundle.pem > star.kumari.net-20110723-bundle.pem 
    

    Now tell Postfix about it:

     # TLS parameters
     smtpd_tls_cert_file=/etc/ssl/certs/star.kumari.net-20110723-bundle.pem 
     smtpd_tls_key_file=/etc/ssl/private/star.kumari.net-20110723.key
     smtpd_use_tls=yes
     smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
     smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

     

  • Mounting a QEMU Image

     

    Mounting a QEMU Image

    In order to mount a QUMU / KVM disk image you need to use qemu-nbd, which lets you use the NBD protocol to share the disk image on the network.

    First you need the module loaded:

     sudo modprobe nbd max_part=8
    

    Then you can share the disk on the network and create the device entries:

     sudo qemu-nbd --connect=/dev/nbd0 file.qcow2
    

    Then you mount it:

    sudo mount /dev/nbd0p1 /mnt/kvm
    

    When done, unmount and unshare it:

    sudo umount /mnt/kvm
    sudo nbd-client -d /dev/nbd0

     

  • Creating a Xen image

    A friend of mine recently lost access to his server and so I figured I’d try and create him a virtual machine on one of my boxes while he was getting his back online.

    “No worries”, think I, “I’ll just install something like KVM and it’ll all be good”. Unfortunately I found out that the CPUs in my server (dual core P4 Xeon 3Ghz in a Dell 2850) don’t support the Intel-VT instruction set needed to run KVM. “Well, ok, I’ll just some something like ‘apt-get install ubuntu-xen-server’ and all will be good”…

     

    Unfortunately it turned out to be more complex than that — here is what I did to get it working, so next time I don’t waste 3 hours poking at this.

     

    The Intrepid xen packages install ‘xen-3.3.gz’ in /boot, but there are there is no kernel or modules file. Turns out the repository doesn’t have those, so you need to do:

    Kernel: http://packages.debian.org/lenny/i386/linux-image-2.6.26-2-xen-686/download

    Modules: http://packages.debian.org/lenny/i386/linux-modules-2.6.26-2-xen-686/download

    Then install using

    sudo dpkg -i linux-image-2.6.26-2-xen-686_2.6.26-15lenny2_i386.deb linux-modules-2.6.26-2-xen-686_2.6.26-15lenny2_i386.deb 

     

    After that I made the virtual machine and started it – “Hmmm, this is too easy…”, and sure enough, it was…

     

    Connecting to the virtual console only gave me:

    starting system log daemon: syslogd.
    Starting kernel log daemon: klogd.
    * Not starting internet superserver: no services enabled.
    Starting OpenBSD Secure Shell server: sshd[    8.032223] NET: Registered protocol family 10
    [    8.032777] lo: Disabled Privacy Extensions
    .
    Starting periodic command scheduler: crond.

     

    and then nothing — I figured I’d try and loging over ssh, but the install process hadn’t prompted me for a root password. I mounted the dis image and added a user, then tried logging in over ssh. Still no luch, I got:

     

    After much prodding I got everyting working correctly, here is how:

     

    The general config is in /etc/xen-tools/xen-tools.conf

    To create the image do:

     root@machine:/etc/xen-tools# sudo xen-create-image --hostname=xen1.example.com --ip xx.xx.xx.xx --ide --force --passwd --role udev
    General Information
    --------------------
    Hostname  : xen1.example.com
    Distribution  : lenny
    Partitions  : swap 128Mb (swap)
    / 4Gb (ext3)
    Image type  : sparse
    Memory size  : 128Mb
    Kernel path  : /boot/vmlinuz-2.6.26-2-xen-amd64
    Initrd path  : /boot/initrd.img-2.6.26-2-xen-amd64
    Networking Information
    ----------------------
    IP Address 1  : xx.xx.xx.xx [MAC: 00:16:3E:xx:xx:xx]
    Netmask  : xx.xx.xx.xx
    Gateway  : xx.xx.xx.xx
    Creating partition image: /home/xen/domains/xen1.example.com/swap.img
    Done
    Creating swap on /home/xen/domains/xen1.example.com/swap.img
    Don
    Creating partition image: /home/xen/domains/xen1.example.com/disk.img
    Done
    Creating ext3 filesystem on /home/xen/domains/xen1.example.com/disk.img
    Done
    Installation method: debootstrap
    ...

    To start the image, do:

     root@machine:/etc/xen-tools# sudo xm create /etc/xen/xen1.example.com.cfg

    To connect to the console, run

    xm console xen1.example.com

    . To disconnect the console, its ^-[ The config file for xen-tools is:

    /etc/xen-tools/xen-tools.conf
    dir = /home/xen
    ...
    #
    ##
    # Disk and Sizing options.
    ##
    #
    size = 4Gb # Disk image size.
    memory = 128Mb # Memory size
    swap = 128Mb # Swap size
    # noswap = 1 # Don't use swap at all for the new system.
    fs = ext3 # use the EXT3 filesystem for the disk image.
    dist = lenny # Default distribution to install.
    image = sparse # Specify sparse vs. full disk images.
    ...
    # Uncomment and adjust these network settings if you wish to give your
    # new instances static IP addresses.
    #
    netmask = xx.xx.xx.xx
    gateway = xx.xx.xx.xx
    ...
    #
    # Uncomment the following line if you wish to interactively setup
    # a new root password for images.
    #
    passwd = 1
    ...
    #
    # If you're using a newer version of the Xen guest kernel you will
    # need to make sure that you use 'xvc0' for the guest serial device,
    # and 'xvdX' instead of 'sdX' for serial devices.
    #
    # You may specify the things to use here:
    serial_device = hvc0