One line web server

The following one line script will create a web server running on port 80 using nc (netcat):

while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; cat index.html; } | nc -l 8080; done

Purge Removed packages

Packages marked as rc by dpkg mean that the configuration files are not yet removed. The following command will purge them:

dpkg --list |grep "^rc" | cut -d " " -f 3 | xargs -r sudo dpkg --purge

Removing hosts from backuppc

Simply remove the host from the web interface and rm -rf the pc/<host> directory, then wait for the next BackupPC_nightly run - it will remove all superfluous files from the pools.

The path for this directory usually is:

/var/lib/Backuppc/pc/<hostname>

If you want to force the clean-up process, you can remove your host like this:

1. Login to the Backuppc server
2. Remove the host in the Backuppc web-interface (under hosts)
3. remove it's directory /var/lib/Backuppc/pc/<hostname>:

rm -rf /var/lib/Backuppc/pc/<hostname>

4. Shutdown backuppc:

service backuppc stop

5. Change into backuppc:

su - backuppc

6. Run the nightly script:

/usr/share/BackupPC/bin/BackupPC_nightly 0 255

7. Go back to root:

exit

8. Start backuppc again:

service backuppc start

Reclaim free space from Time Machine sparsebundle

You msut run these commands as root:

sudo su - 

Make sure the mount point exists:

mkdir -p /Volumes/TM

Then mount the afp share:

mount_afp 'afp://user:password@afp_server_address/share_name' /Volumes/TM

Now use hdiutil to reclaim the available free space:

hdiutil compact /Volumes/TM/ComputerName.sparsebundle/

unmont the share:

umount /Volumes/TM/

If you get an error message saying:

hdiutil: compact failed - Resource temporarily unavailable

You must make sure you don't have the afp share mounted elsewhere, you can check your mounts with:

df -h

If the output contains a line with your afp server's address or with the string "Time Machine" you have to unmount them.

The following script will do all that for you:

SRV="afp_server_address"
SAVEIFS=$IFS
IFS=$'\n';
for v in $(df -h | grep -E "$SRV|Time\sMachine"  | cut -d"%" -f3 |  sed -e "s/ *\//\//"); do
    umount "$v"
done
IFS=$SAVEIFS
mkdir -p /Volumes/TM
mount_afp "afp://user:password@$SRV/share" /Volumes/TM
hdiutil compact "/Volumes/TM/$(scutil --get ComputerName).sparsebundle"
umount /Volumes/TM/

Changing Time Machine Backup Interval

You can use the following command to change the Time Machine backup interval:

sudo defaults write /System/Library/LaunchDaemons/com.apple.backupd-auto StartInterval -int 14400

The time interval is in seconds, so 43200 will start a backup every 12hrs.

Checkout my previous post to learn how to manually delete Time Machine backups.

Manage time machine backups

Some times you get some errors saying that your time machine's disk is full and Time Machine could not complete the backup so you need to  manually delete old backups.

tmutil provides methods of controlling and interacting with Time Machine, as well as examining and manipulating Time Machine backups. Common abilities include restoring data from backups, editing exclusions, and comparing backups.

tmutil latestbackup

Will output the path to the most recent backup and

tmutil listbackups

will list all existing backups, if you use the same backup disk for multiple machines, you can get just the backups from your machine with:

tmutil listbackups | grep "$(scutil --get ComputerName)"

The following command will delete the backups from a mac named old_mac_name:

sudo tmutil delete /Volumes/drive_name/Backups.backupdb/old_mac_name

If you want to be safe, you can pick one snapshot to delete first to be sure the command works as intended. This is nice since it could take hours to clean up some larger backup sets and you want to leave the Mac confident it's deleting the correct information store.

You can use the tmutil tool to delete backups one by one.

sudo tmutil delete /Volumes/drive_name/Backups.backupdb/mac_name/YYYY-MM-DD-hhmmss

Since tmutil was introduced with Lion, this will not work on earlier OS versions.

The tmutil delete command only removes the backup from the sparse bundle. It doesn’t actually free the disk space. To do that, you have to go a little deeper.

On your Mac is a mount point called /Volumes. You can examine the contents of this mount point with ls:

cd /Volumes
ls -1

Should output something like:

Macintosh HD
Recovery HD
Time Machine Backups
TimeMachine

These are the names of all the mounted disks (or things that look like disks) on your Mac. Notice two likely candidates for your actual TimeMachine volume. Yours may be named slightly differently, but the one you want is the one that actually shows files of type .sparsebundle . In my case, it is the volume TimeMachine:

sudo ls -l TimeMachine/

and you should see something similar to:

...
drwxr-x---@ 1 root wheel 264 Jul 25 08:21 sysadmin’s MacbookPro.sparsebundle
...

Notice that you don’t actually own the file. (Had I not used the sudo command with ls I could not have listed the contents of /Volumes/TimeMachine)

That .sparsebundle file for your Mac is where all your backup sets live. TimeMachine manages the contents of this file, but doesn’t do anything automatically to reduce its size. Luckily there is another tool for that, but you’ll have to be root to run it:

sudo su -
hdiutil compact /Volumes/TimeMachine/YourBackup.sparsebundle

Sample output:

Starting to compact…
Reclaiming free space…
...................................................
Finishing compaction…
Reclaimed 3.1 GB out of 304.1 GB possible.

That’s it! In this example I reclaimed 3.1GB of actual disk space on my TimeMachine volume. 

The following bash script will remove the oldest backup and reclaim the free space:

COMPUTER_NAME=$(/usr/sbin/scutil --get ComputerName)
NBACKUPS=$(/usr/bin/tmutil listbackups | /usr/bin/grep "$COMPUTER_NAME" | /usr/bin/wc -l)
OLDEST_BACKUP=$(/usr/bin/tmutil listbackups | /usr/bin/grep "$COMPUTER_NAME" | /usr/bin/head -n1)
LATEST_BACKUP=$(/usr/bin/tmutil latestbackup)
echo Latest backup: $LATEST_BACKUP
if [[ -n "$LATEST_BACKUP" && "$LATEST_BACKUP" != "$OLDEST_BACKUP" ]]
then
    echo "$NBACKUPS backups. Delete oldest: ${OLDEST_BACKUP##*/} [y/N]? \c"
    read answer
    case $answer in
   y*)
  echo Running: /usr/bin/sudo /usr/bin/tmutil delete "$OLDEST_BACKUP"
  /usr/bin/sudo time /usr/bin/tmutil delete "$OLDEST_BACKUP"
    echo "Do you wish to reclaim the free space now? [y/N]? \c"
    read answer
    case $answer in
    y*)
     mkdir -p /Volumes/TM
     mount_afp 'afp://user:pass@afp_server_address/share_name' /Volumes/TM
     hdiutil compact "/Volumes/TM/$(scutil --get ComputerName).sparsebundle"
     umount /Volumes/TM/
    ;;
    *)
 echo No change
    ;;
        esac
   ;;
   *)
  echo No change
   ;;
    esac
else
    echo "No backup available for deletion"
fi

In the script above, don't forget to change the afp URL (afp://user:pass@afp_server_address/share_name) to your own.

Sorry, Command-not-found Has Crashed

When you try to execute a command that is not installed Ubuntu tries to hint you on the package that you should install but some times, especially after an upgrade, you get an error message saying:

Sorry, command-not-found has crashed! Please file a bug report at:
(...)

This solves the problem:

export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales

Ubuntu as Time Machine server

This guide will help you to install and configure the netatalk servise on an Ubuntu server so it can function as a Time Machine backup server for your Mac OS machines.

First install the necessary packages:

sudo apt-get install netatalk avahi-daemon libnss-mdns

Open the netatalk default configuration file:

sudo vi /etc/default/netatalk

Modify the lines:

ATALKD_RUN=yes
PAPD_RUN=no
CNID_METAD_RUN=yes
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

Edit the atalkd.conf file:

sudo vi /etc/netatalk/atalkd.conf

add to the bottom

eth0

Edit the AppleVolumes.default file:

sudo vi /etc/netatalk/AppleVolumes.default

add to the bottom:

/backups/timemachine "Time Machine" allow:@admin cnidscheme:cdb volsizelimit:200000 options:usedots,upriv,tm

The example above also limits the size shown to OS X as 200 GB (the number is given in MiB, so it's 200,000 times 1024 in the real world)

Edit the afpd configuration file:

sudo vi  /etc/netatalk/afpd.conf

add to the bottom:

- -transall -uamlist uams_dhx.so,uams_dhx2.so -nosavepassword -advertise_ssh -mimicmodel TimeCapsule6,106 -setuplog "default log_warn  /var/log/afpd.log"

Create a configuration file for the avahi afpd discovery:

sudo vi  /etc/avahi/services/afpd.service

and enter the following into it:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
 <name replace-wildcards="yes">%h</name>
 <service>
 <type>_afpovertcp._tcp</type>
 <port>548</port>
 </service>
 <service>
 <type>_device-info._tcp</type>
 <port>0</port>
 <txt-record>model=MacPro</txt-record>
 </service>
</service-group> 

Restart the services:

sudo service netatalk restart
sudo service avahi-daemon restart

Test imap using telnet

For added security, you can encrypt your IMAP connection. This requires that your server supports SSL or TLS and that you have access to an SSL/TLS client program, for example OpenSSL, to use instead of telnet.

As the port-number normally is 993, an example OpenSSL command would be openssl s_client -connect imap.example.com:993 -quiet. (If you would like to see the public key of the server, as well as some other encryption-related information, omit -quiet.) The server should then start an IMAP session, displaying a greeting such as the * OK Dovecot ready example below.

telnet imap.example.com 143

#output: Trying 193.136.28.29...
#output: Connected to imap.example.com.
#output: Escape character is '^]'.
#output: * OK Dovecot ready.

a1 LOGIN MyUsername MyPassword
#output: a1 OK Logged in.

a2 LIST "" "*"
#output: * LIST (\HasNoChildren) "." "INBOX"
#output: a2 OK List completed.

a3 EXAMINE INBOX
#output: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
#output: * OK [PERMANENTFLAGS ()] Read-only mailbox.
#output: * 1 EXISTS
#output: * 1 RECENT
#output: * OK [UNSEEN 1] First unseen.
#output: * OK [UIDVALIDITY 1257842737] UIDs valid
#output: * OK [UIDNEXT 2] Predicted next UID
#output: a3 OK [READ-ONLY] Select completed.

a4 FETCH 1 BODY[]
#output: * 1 FETCH (BODY[] {405}
#output: Return-Path: sender@example.com
#output: Received: from client.example.com ([192.0.2.1])
#output:    by mx1.example.com with ESMTP
#output:    id <20040120203404.CCCC18555.mx1.example.com@client.example.com>
#output:    for <recipient@example.com>; Tue, 20 Jan 2004 22:34:24 +0200
#output: From: sender@example.com
#output: Subject: Test message
#output: To: recipient@example.com
#output: Message-Id: <20040120203404.CCCC18555.mx1.example.com@client.example.com>
#output:
#output: This is a test message.
#output: )
#output: a4 OK Fetch completed.

a5 LOGOUT
#output: * BYE Logging out
#output: a5 OK Logout completed.

Finding external IP using the command line

The easiest way is to use an external service via a commandline browser or download tool. Since wget is available by default in Ubuntu, we can use that.

To find your ip, use:

wget -qO- http://ipecho.net/plain ;

You can do the same using curl:

curl ipecho.net/plain ; echo

How to test a listening TCP/UDP port through nc

Netcat (nc) can also be used for a lot of other purposes. It can also be used as a very fast basic port scanner, you can scan a port or a range.

To scan a range of UDP ports 1 to 1000:

nc -vzu destination_ip 1-1000

To scan a range of TCP ports 1 to 1000

nc -vz destination_ip 1-1000

Using the IP command

The command /bin/ip has been around for some time now. But people continue using the older command /sbin/ifconfig. ifconfig won't go away quickly, but its newer version, ip, is more powerful and will eventually replace it.
So here are the basics of the new ip command.

Assign a IP Address to Specific Interface:

sudo ip addr add 192.168.50.5 dev eth1 

Check an IP Address:

sudo ip addr show 

Remove an IP Address:

sudo ip addr del 192.168.50.5/24 dev eth1 

Enable Network Interface:

sudo ip link set eth1 up 

Disable Network Interface:

sudo ip link set eth1 down 

 Check Route Table:

sudo ip route show 

Add Static Route:

sudo ip route add 10.10.20.0/24 via 192.168.50.100 dev eth0 

Remove Static Route:

sudo ip route del 10.10.20.0/24 

Add Default Gateway:

sudo ip route add default via 192.168.50.100

Change Root DN Password on OpenLDAP

First, we need to locate the credentials information of the administrator account in the correct database within the LDAP tree.

This can be done using the ldapsearch command:

ldapsearch -LLL -Y EXTERNAL -H ldapi:/// -b  cn=config olcRootDN=cn=admin,dc=example,dc=com dn olcRootDN olcRootPW

(replace the olcRootDN value with the correct value to match your configuration)

This command will return something like:

SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
dn: olcDatabase={1}hdb,cn=config
olcRootDN: cn=admin,dc=example,dc=com
olcRootPW: {SHA}ks1xBVfgRXavGCpkPefc9hRHL4X=

There are two interesting information we know now:

we need to modify the entry “dn: olcDatabase={1}hdb,cn=config“
the current password is hashed with SHA1 algorythm.
To generate our new password with the same algorythm we'll use the command slappasswd with the syntax:

slappasswd -h <the hashing scheme we want to use - for example {SHA}>

The system will then prompt you for the new password to use, twice, and will finally display the hashed value we’re interested in:

root@testbox:~# slappasswd -h {SHA}
New password:
Re-enter new password:
{SHA}W6ph5Mm7Ps6GglULbPgzG37mj0g=

Then we’ll proceed to modify the entry we’ve identified above using the command:

root@testbox:~# ldapmodify -Y EXTERNAL -H ldapi:///

The system will start the listening mode for modifying commands:

SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0

First, we enter the entry we want to modify:

dn: olcDatabase={1}hdb,cn=config

Second, we type in the parameter we want to modify:

replace: olcRootPW

Third, we type in the new password generated above (copy and paste is MUCH less error prone than manual typing at this point ;) )

olcRootPW: {SHA}W6ph5Mm7Ps6GglULbPgzG37mj0g=

Hit Enter another time to commit the modification and the following line will appear:

modifying entry "olcDatabase={1}hdb,cn=config"

After this, you can exit the listening mode with CTRL+C and restart the LDAP database service using:

service slapd stop
service slapd start

and login now with the new password set.

Linux Stress tests

Consume CPU:

Fork bomb:

:(){ :|:& };:

The next one will load four CPU cores at 100%:

for i in `seq 1 4` ; do while : ; do : ; done & ; done

Or:

for i in `seq 1 4` ; do cat /dev/zero > /dev/null & ; done

Or:

#!/bin/bash
duration=120 # seconds
instances=4 # cpus
endtime=$(($(date +%s) + $duration))
for ((i=0; i<instances; i++))
do
while (($(date +%s) < $endtime)); do : ; done &
done

Using the stress tool:

stress --cpu 3

Consume RAM:

Create a 30gb ramdisk and fills it with file full of zeroes:

sudo mount -t tmpfs -o size=30G tmpfs /mnt
dd if=/dev/zero of=/mnt/tmp bs=10240 count=30720MB

Create a giant virable:

x="x" ; while : ; do x=$x$x ; echo -n "." ; done

Consume Disk:

dd if=/dev/zero of=bigfile bs=10240 count=30720MB

Simulate packet loss:

For randomly dropping 10% of incoming packets:

iptables -A INPUT -m statistic --mode random --probability 0.1 -j DROP

and for dropping 10% of outgoing packets:

iptables -A OUTPUT -m statistic --mode random --probability 0.1 -j DROP

Calculating total disk usage by files with specific extension

For example if you want to check how much space is being used by log files on your entire system, you can use the following:

find / -type f -name "*.log*" -exec du -b {} \; | awk '{ sum += $1 } END { kb = sum / 1024; mb = kb / 1024; gb = mb / 1024; printf "%.0f MB (%.2fGB) disk space used\n", mb, gb}'

Just replace "*.log*" with the file extension you want to search for and the above will give you the disk used by the sum of all the files with that extension.

Encrypt and decrypt files using openssl

Here's a safe way to pass sensitive files over email...

To encrypt use:

openssl enc -e -bf-cbc -in <FILE.zip> -out <FILE.ENC>

This will ask you for a password that you'll need to decrypt the file.

And to decrypt:

openssl enc -d -bf-cbc -in <FILE.ENC> -out <FILE.zip>

Rename files from upper case filename to lower case

The following one line script will rename every file (in the current folder) to lowercase:

for i in *; do mv $i `echo $i | tr [:upper:] [:lower:]`; done

tail -f with highlighting

If you want to highlight something when doing ‘tail -f’ you can use the following command:

tail -f /var/log/logfile | perl -p -e 's/(something)/\033[7;1m$1\033[0m/g;'

or if your terminal supports colours, e.g. linux terminal, you can use this:

tail -f /var/log/logfile | perl -p -e 's/(something)/\033[46;1m$1\033[0m/g;'

If you need to highlight multiple words you can use something like this:

tail -f /var/log/logfile | perl -p -e 's/\b(something|something_else)\b/\033[46;1m$1\033[0m/g;'

and if you want it to beep on a match use this:

tail -f /var/log/logfile | perl -p -e 's/(something)/\033[46;1m$1\033[0m\007/g;'

If you find that perl is too heavy for this you can use sed:

tail -f /var/log/logfile | sed "s/\(something\)/\x1b[46;1m\1\x1b[0m/g"

Note, that in the last example you have to actually type “cntl-v cntl-[” in place of “^[”
\x1b character can also be used as the escape character.

For the full list of control characters on Linux you can look at:

man console_codes

Show the 20 most CPU/Memory hungry processes

Display the top 20 running processes - sorted by memory usage
ps returns all running processes which are then sorted by the 4th field in numerical order and the top 20 are sent to STDOUT.

ps aux | sort -nk +4 | tail -20

Show the 20 most CPU/Memory hungry processes
This command will show the 20 processes using the most CPU time (hungriest at the bottom).

ps aux | sort -nk +3 | tail -20

Or, run both:

echo "CPU:" && ps aux | sort -nk +3 | tail -20 && echo "Memory:" && ps aux | sort -nk +4 | tail -20

View process tree

One way to get the current process tree is to use the PS command, like this:

ps faux

Another way is to use the command pstree which will give you a nicer output, like this:

pstree -l -a

the -l option enables the "long lines", by default lines will be truncated and the -a option is for pstree to show the command line arguments of each process. There are other options that you can use, like the -p which will display the IDs of each process.

If you want to see the tree of a particular process you can pass the process PID to pstree:

pstree -l -a 5567

If you don't know the PID of the process you want you can use the following method:

pstree -l -a $(pidof cron)

This will display cron and all of it's children.

You may also see the process tree of a particular user:

pstree -l -a root