[{"categories":null,"contents":"Looking to track new releases of your favorite open source software? There are multiple ways to track new versions, including using RSS!\nIs your project or favorite missing? That is most likely easily fixed by sharing it.\n","permalink":"https://linux-audit.com/resources/open-source-software-releases/","tags":["rss"],"title":"Open source software releases"},{"categories":null,"contents":"Looking to follow more Linux RSS feeds? Then this might be the overview that you are looking for. It\u0026rsquo;s work in progress and relevant RSS feeds are being collected. Is yours missing? That is most likely easily fixed by sharing it.\nWhy this RSS list? The web changes and so do search engines. Good up-to-date Linux blogs are sometimes hard to find. That\u0026rsquo;s a shame and also a risk, especially with AI filling up some of the space. High-quality articles and real experiences are what make the web great! That is also the reason why this initiative has started to collect and categorize Linux blogs and the related RSS feeds. Each of the blogs are reviewed and a related data is populated.\nThe requirements to get listed:\nRSS feed (RSS / Atom) Cover Linux or at least sometimes Topics such as networking, security, and virtualization are appreciated as well Notes The current listing is boring and simple. The primary focus is now on adding the feeds, then slowly populating additional meta information about each entry. At a later stage checks will be added to see if a blog is still up-to-date. It not, it might be delisted. Where possible, the content creator will be contacted.\n","permalink":"https://linux-audit.com/resources/linux-rss-feeds/","tags":["rss"],"title":"Linux RSS feeds"},{"categories":["Nginx","Web"],"contents":"Nginx has many options, with one of them the option to rate limit requests. This is a very helpful option ensure HTTP clients behave themselves a little bit. If they don\u0026rsquo;t, they can be quickly discovered and actions taken. In this article we have a look at how to set this up.\nSetting up a rate limited zone The first step it so set up a zone where rate limiting is applied. With nginx we need to do this in a few steps.\nMatch IP address to determine if a client is restricted or unrestricted Map the IP address to leverage its binary notation of the IP address Define the rate limited zone Optionally define the error code to use Apply zone in virtual host Steps 1-4 can be defined within the http context, so that we can use the rate limiting functionality in multiple virtual hosts. This is normally done in your main nginx configuration file (nginx.conf).\nThe first steps are split to give a little bit more insights, with a full configuration piece at the end of the section.\nDefine geo First we define if a client needs to be rate limited or not. We use the variable $unrestricted_ip for this. If it contains a zero (0), then it is a normal client that should be rate limited. If we define the value 1 to it, then no rate limiting is applied.\nhttp { # ... other configuration options ... # Define if a client IP is unrestricted (by default it is not) geo $unrestricted_ip { default 0; # Default set to 0, normal clients are definitely not unrestricted proxy 1.2.3.4/24; # Use this if traffic comes in via a proxy, otherwise remove proxy abc:abc:abcd::/48; # Same, but for IPv6 ::1 1; # localhost 1.2.3.4/24 1; # Our proxies should never be limited 66.249.64.0/19 1; # We allow Google bot to go nuts 127.0.0.1 1; # localhost } } In this example we defined our default (0), we added proxies (optional), and defined some hosts and network ranges that should not be rate limited.\nMapping restricted clients for accounting purposes Next step is to store the IP addresses of normal client in a binary format, so that nginx can process them more efficiently. This is done by using the value from $binary_remote_addr, which is a built-in variable. We tell nginx that if a client is restricted, that the binary representation of the client IP address should be used for accounting purposes. If the client is unrestricted, then empty the value. The result will be stored in variable $restricted_ip_key.\nhttp { geo $unrestricted_ip { ...; } # Next step: map client IP to the binary notation of the IP address, otherwise empty it map $unrestricted_ip $restricted_ip_key { 0 $binary_remote_addr; # key set, so rate-limiting applies 1 \u0026#39;\u0026#39;; } } Create a rate limited zone Now it is time to create the zone itself. You can give it any name that you want, for this example we call it ratelimitedzone.\nFor this zone we reserve 10 megabytes of memory, enough to hold a lot of IP addresses. If you have a very busy server, tune this to your needs.\nThe rate limit is defined with 60 requests per minute, which translated to 1 every second. That is not much, especially considering that HTTP clients typically make multiple requests shortly to pull in the HTML, CSS, JavaScript, fonts, etc. So this needs some fine-tuning when we apply the zone, by allowing a \u0026lsquo;burst\u0026rsquo; of requests. If your average web page has around 15 requests, this rate limit of 60 would mean the visitor could browse to 4 pages within 1 minute. Typically a visitor takes the time to read the page and won\u0026rsquo;t switch that quickly between multiple pages.\nThe last line defines what error code we want to return. Unfortunately, nginx returns by default a HTTP 503 error, indicating it is a server issue. There is a better response code and that is HTTP 429 or Too Many Requests, exactly what happens when a client performs too many requests in some amount of time. So we use that status code when rate limiting is active.\nhttp { geo $unrestricted_ip { ...; } map $unrestricted_ip $restricted_ip_key { ...; } # Define our zone with the binary notation of the IP address limit_req_zone $restricted_ip_key zone=ratelimitedzone:10m rate=60r/m; limit_req_status 429; # Return HTTP/429 = Too Many Requests (instead of default 503) } Full example of the rate limited zone So if we combine these parts, you get something like this:\nhttp { # ... other configuration options ... # Define if a client IP is unrestricted (by default it is not) geo $unrestricted_ip { default 0; # Default set to 0, normal clients are definitely not unrestricted proxy 1.2.3.4/24; # Use this if traffic comes in via a proxy, otherwise remove proxy abc:abc:abcd::/48; # Same, but for IPv6 ::1 1; # Do not restrict localhost 1.2.3.4/24 1; # Our proxies should never be limited 66.249.64.0/19 1; # We allow Google bot to go nuts 127.0.0.1 1; # Do not restrict localhost } # Next step: map client IP to the binary notation of the IP address, otherwise empty it map $unrestricted_ip $restricted_ip_key { 0 $binary_remote_addr; # key set, so rate-limiting applies 1 \u0026#39;\u0026#39;; } # Define our zone with the binary notation of the IP address limit_req_zone $restricted_ip_key zone=ratelimitedzone:10m rate=60r/m; limit_req_status 429; # Return HTTP/429 = Too Many Requests (instead of default 503) } This configuration could be simplified by just using the last two lines (with $binary_remote_addr). The downside of that is that all HTTP clients will be rate limited. That may get you into troubles if you have a fairly strict rate defined and \u0026ldquo;good\u0026rdquo; clients get rate limited as well. For example, the Google bot typically behaves well, but now and then it wants to do a quick update of many pages and may hit the rate limits. Also you might have services like a link checker that you want to grant full speed. For this reason, it is useful to allow exceptions for those systems.\nConfiguring the virtual host With this initial piece of configuration, the next step is applying this configuration within a virtual host. This way the zone will actually be used.\nserver { listen ...; server ...; # Apply our rate limited zone which is defined within the HTTP context (in /etc/nginx/nginx.conf) limit_req zone=ratelimitedzone burst=90 nodelay; } Within our virtual host configuration we have now defined that we want to apply rate limiting. We refer to our zone (ratelimitedzone) and can give it some additional options. In this case we allow a burst, meaning that clients can temporarily go beyond the related of 60 requests per minute (= 1 per second). This is useful to allow legitimate clients to pull in all required files for the first time (e.g. CSS, JavaScript) to serve a page. In this example we will not delay any requests (nodelay). See the related module for details about fine-tuning this to your needs.\nNot sure if your chosen rate limit is correct to properly catch bad-behaving clients while still allowing legitimate clients? Use the limit_req_dry_run option to do accounting and logging, but not enforcing the limits.\nlimit_req_dry_run on; Restart and testing After implementing the changes, confirm that your nginx configuration looks fine with the nginx command and the -t option:\nnginx -t\nAll good? Restart the server, for Linux that is usually done with systemctl.\nsystemctl restart nginx.service\nNext step is testing the rate limit. You could do this by setting the rate limit values very low and send some requests with curl. Another option is to fire up the ab tool (Apache Bench) and do a short stress test.\nBlocking repeating offenders If you have repeating offenders, then they will be showing up with a 429 status code in the log file. If a client receives this message and back offs, then it actually understood that it was going too quick. But some clients simply ignore it and will continue to perform their requests. This is especially the case for crawlers that want to index as much pages as quickly as possible.\nOne option is to consider to block the IP addresses that keep hitting the rate limit values. One option is to count the number of 429s per IP address and it goes over a threshold, to block them.\nNeed more ideas how to do this? Let it know!\n","permalink":"https://linux-audit.com/web/nginx-rate-limit-http-clients-by-number-of-requests/","tags":["howto","nginx"],"title":"Rate limit HTTP clients with nginx"},{"categories":["System administration"],"contents":"The uname command provides basic information about a Linux system and the related hardware. It can show the architecture of the machine and the processor.\n","permalink":"https://linux-audit.com/system-administration/commands/uname/","tags":["cpu","hardware","hostname","kernel","linux","hardware","tools"],"title":"uname: show basic system information"},{"categories":["System administration"],"contents":"The head command provides the first 10 lines of a file to the standard output. It can read multiple files or from the standard input, such as using pipes.\n","permalink":"https://linux-audit.com/system-administration/commands/head/","tags":["data processing","linux","tools"],"title":"head: show first number of lines from a file"},{"categories":["System administration"],"contents":"The pidof command can return the process ID (PID) for a process similar to pgrep. It is a very useful tool to create one-liners where one command needs to take an action on a PID, but the you only have the name available.\n","permalink":"https://linux-audit.com/system-administration/commands/pidof/","tags":["processes"],"title":"pidof: retrieve PID when searching for process names"},{"categories":["System administration"],"contents":"The lsns command is used for showing information about Linux namespaces. It is part of the util-linux package, which comes with many tools related to system administration. By just running the command most details are already presented. It is a good way to learn what processes are using a different namespace than the global ones.\nUsed columns The command uses several columns to display the information. These are the columns and a description of what information it represents.\nColumn Description COMMAND Command line of the PID NPROCS Number of processes inside the namespace NETNSID Namespace ID for network subsystem NS Namespace identifier by inode number NSFS Mountpoint of nsfs virtual filesystem ONS Owner namespace identifier (inode number) PATH Path to the namespace PID Lowest process ID within the namespace PNS Parent namespace identifier (inode number) PPID Parent process ID (PPID) of the proces ID (PID) TYPE Namespace type (cgroup, ipc, mnt, net, pid, time, user, uts) UID User ID of the PID USER Username of the PID ","permalink":"https://linux-audit.com/system-administration/commands/lsns/","tags":["linux","namespaces","tools"],"title":"lsns: show active Linux namespaces"},{"categories":["System administration"],"contents":"The dmesg command can be used to show the contents of the Linux kernel ring buffer and important kernel logging. It includes events about the boot process, hardware, systemd, segmentation faults of processes, and more.\n","permalink":"https://linux-audit.com/system-administration/commands/dmesg/","tags":["linux","tools"],"title":"dmesg: show log events from kernel ring buffer"},{"categories":null,"contents":"The kernel ring buffer on Linux holds important system events and makes it available to the system administrator. It is maintained by the kernel itself and is stored in memory. The contents of the ring buffer is available to user space, so it can easily be viewed with a tool like dmesg. The kernel will log events to the buffer that are usually related to the boot process, kernel modules being loaded or unloaded, hardware support, systemd activation, firewall events, memory, and serious issues with processes.\nBuffer size The size of the kernel ring buffer is defined during compilation of the kernel itself, so typically your Linux distribution decides an acceptable size. To find this, we need to extract the value of CONFIG_LOG_BUF_SHIFT from the kernel configuration file. This file is typically stored in /boot or /proc.\n# grep ^CONFIG_LOG_BUF_SHIFT /boot/config-$(uname -r) CONFIG_LOG_BUF_SHIFT=17 If the configuration is not available in the /boot, then it might be stored in /proc.\nzgrep CONFIG_LOG_BUF_SHIFT /proc/config.gz\nThis will return a value, like the value 17 above. To turn this into a size, we need to calculate this by using 2 to the power of 17 (2^17), resulting in the number 131072, which means 128 kilobytes.\nValue Size of buffer 18 256 KB 17 128 KB 16 64 KB 15 32 KB 14 16 KB 13 8 KB 12 4 KB ","permalink":"https://linux-audit.com/what-is/kernel-ring-buffer/","tags":["kernel","linux","logging","what-is"],"title":"What is a kernel ring buffer?"},{"categories":["System administration"],"contents":"The lsfd command can be used to show open file descriptors for the system or a selection. The project states it is intended to be a replacement of the lsof command, although it is not a drop-in replacement. This means the parameters and syntax will be different. One of the major benefits of lsfd is that it is purely aimed at Linux, so it has better support for Linux features, like cgroups.\nThe initial commit for lsfd within the util-linux project was at Oct 6, 2021.\n","permalink":"https://linux-audit.com/system-administration/commands/lsfd/","tags":["file system","linux","tools"],"title":"lsfd command"},{"categories":["System administration"],"contents":"The numactl tool can be used to define a specific NUMA policy related to scheduling and memory placement. The given policy will be applicable to the process and its children.\n","permalink":"https://linux-audit.com/system-administration/commands/numactl/","tags":["linux","memory","processes","tools"],"title":"numactl: control NUMA policy for processes and shared memory"},{"categories":["Kernel"],"contents":"The sysctl key kernel.sched_schedstats is used to enable statistics of the Linux scheduler. When enabled, more detailed information will be stored within the /proc directory and specifically in the sched file. To see the information look at the path /proc/PID/sched. See also the explanation of values in /proc/PID/sched for more details.\n","permalink":"https://linux-audit.com/kernel/sysctl/kernel/kernel.sched_schedstats/","tags":["kernel","linux","scheduler","statistics","sysctl"],"title":"kernel.sched_schedstats"},{"categories":["Kernel","Linux"],"contents":"Introduction The Linux scheduler handles task scheduling. It is one of the most important jobs of the kernel and consists of deciding which thread can run and for how long. This should be done in such a way so that each of them gets enough time on the CPU. To ensure important processes get more priority, the scheduler uses a runqueue and defines for each task how much priority it gets.\nTracking statistics about tasks Also part of the Linux scheduler is tracking processes and providing statistics in the form of sched files in the /proc directory. This is useful for developers and system administrators to troubleshoot any issues. That is why for each process ID (PID) there is a file available, with specifics for that PID. So for the first PID the related file would be /proc/1/sched. But what do these statistics mean? Although the Linux kernel has its source code available, understanding the details is definitely not easy for the average system administrator. This article is trying to sched light (pun intended) light on the file structure and its contents.\nShow details and enabling more statistics To see the scheduler information for a process ID, simply look in the related file.\n# cat /proc/1/sched systemd (1, #threads: 1) ------------------------------------------------------------------- se.exec_start : 46944911.963339 se.vruntime : 155.103393 se.sum_exec_runtime : 566.369810 se.nr_migrations : 56 nr_switches : 2685 nr_voluntary_switches : 2334 nr_involuntary_switches : 351 se.load.weight : 1048576 se.avg.load_sum : 108 se.avg.runnable_sum : 110592 se.avg.util_sum : 110592 se.avg.load_avg : 0 se.avg.runnable_avg : 0 se.avg.util_avg : 0 se.avg.last_update_time : 46944911963136 se.avg.util_est.ewma : 9 se.avg.util_est.enqueued : 0 policy : 0 prio : 120 clock-delta : 36 mm-\u0026gt;numa_scan_seq : 0 numa_pages_migrated : 0 numa_preferred_nid : -1 total_numa_faults : 0 current_node=0, numa_group_id=0 numa_faults node=0 task_private=0 task_shared=0 group_private=0 group_shared=0 The list consists of already a good amount of information. To enable even more detailed statistics, we can active this using the sysctl by setting kernel.sched_schedstats=1.\nsysctl kernel.sched_schedstats=1\nThe number of available fields increased by a lot.\n# cat /proc/1/sched systemd (1, #threads: 1) ------------------------------------------------------------------- se.exec_start : 46944911.963339 se.vruntime : 155.103393 se.sum_exec_runtime : 566.369810 se.nr_migrations : 56 sum_sleep_runtime : 0.000000 sum_block_runtime : 0.000000 wait_start : 0.000000 sleep_start : 46944911.963339 block_start : 0.000000 sleep_max : 0.000000 block_max : 0.000000 exec_max : 0.110262 slice_max : 0.000000 wait_max : 0.000000 wait_sum : 0.000000 wait_count : 1 iowait_sum : 0.000000 iowait_count : 0 nr_migrations_cold : 0 nr_failed_migrations_affine : 0 nr_failed_migrations_running : 0 nr_failed_migrations_hot : 0 nr_forced_migrations : 0 nr_wakeups : 1 nr_wakeups_sync : 0 nr_wakeups_migrate : 0 nr_wakeups_local : 0 nr_wakeups_remote : 1 nr_wakeups_affine : 0 nr_wakeups_affine_attempts : 1 nr_wakeups_passive : 0 nr_wakeups_idle : 0 avg_atom : 0.210938 avg_per_cpu : 10.113746 nr_switches : 2685 nr_voluntary_switches : 2334 nr_involuntary_switches : 351 se.load.weight : 1048576 se.avg.load_sum : 108 se.avg.runnable_sum : 110592 se.avg.util_sum : 110592 se.avg.load_avg : 0 se.avg.runnable_avg : 0 se.avg.util_avg : 0 se.avg.last_update_time : 46944911963136 se.avg.util_est.ewma : 9 se.avg.util_est.enqueued : 0 policy : 0 prio : 120 clock-delta : 27 mm-\u0026gt;numa_scan_seq : 0 numa_pages_migrated : 0 numa_preferred_nid : -1 total_numa_faults : 0 current_node=0, numa_group_id=0 numa_faults node=0 task_private=0 task_shared=0 group_private=0 group_shared=0 So with even more fields, let\u0026rsquo;s have a look at some of the most important fields and values.\nFields and values clock-delta The clock-delta value is displayed when statistics are enabled. The value is the difference between checks of retrieving the CPU clock value. The delta is stored and displayed.\nFile: kernel/sched/debug.c #undef P_SCHEDSTAT { unsigned int this_cpu = raw_smp_processor_id(); u64 t0, t1; t0 = cpu_clock(this_cpu); t1 = cpu_clock(this_cpu); __PS(\u0026#34;clock-delta\u0026#34;, t1-t0); } sched_show_numa(p, m); } iowait_count Each time a process is waiting for a block device, such as a disk to become available, the counter of iowait_count is increased. Together with iowait_sum this provides insights on how often and long a process it waiting.\niowait_sum When a process a waiting on a block device, such as a disk, the time is recorded. A process having a high iowait_sum value, means that is waiting often (or long) for IO to become available. Value is expressed in nanoseconds.\nmm-\u0026gt;numa_scan_seq The field mm-\u0026gt;numa_scan_seq is related to memory management. This value stores a completed scan sequence that is related to NUMA balancing.\nnuma_preferred_nid The value of numa_preferred_nid refers to the preferred node ID of NUMA . The available NUMA nodes can be shown using numactl with the --hardware option.\n# numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 63970 MB node 0 free: 1696 MB node distances: node 0 0: 10 An alternative command is numastat, which also shows the available nodes, including a few statistics.\n# numastat node0 numa_hit 973528 numa_miss 0 numa_foreign 0 interleave_hit 784 local_node 973528 other_node 0 policy The policy refers to the scheduling policy. Typically the value 0 wil be seen, which is the SCHED_NORMAL policy, also referred to as SCHED_OTHER.\nNumber Policy 0 SCHED_NORMAL 1 SCHED_FIFO 2 SCHED_RR 3 SCHED_BATCH 4 Reserved 5 SCHED_IDLE 6 SCHED_DEADLINE 7 SCHED_EXT Policy with number 4 is reserved for SCHED_ISO.\nRelated file: include/uapi/linux/sched.h prio The prio field shows the kernel priority of a process. Usually it shows the value 120, unless the priority has been changed with a command like renice. Setting the nice value to 10, means the value of \u0026lsquo;prio\u0026rsquo; will go to 130. The lower the number, the higher its priority within the runqueue that gives CPU time to tasks.\nScheduler policy Return value Kernel priority User priority (nice value) SCHED_NORMAL, SCHED_BATCH, SCHED_IDLE 0 \u0026hellip; 39 100 \u0026hellip; 139 -20 \u0026hellip; 19 (default 0) SCHED_FIFO, SCHED_RR -2 \u0026hellip; -100 98 \u0026hellip; 0 1 \u0026hellip; 99 SCHED_DEADLINE -101 -1 0 For most processes the policies SCHED_NORMAL (also known as SCHED_OTHER) is used. With a nice value of 0, this means it is in the middle of the range 100-139, which is 120.\nse.sum_exec_runtime The value of se.sum_exec_runtime is the time spent on the CPU and is expressed in nanoseconds.\nse.vruntime The se.vruntime value presents a weighted time (in nanoseconds) a task has run on the CPU. Virtual runtime is used to decide which task deserves to run and is calculated using the task priority, nice value, applicable cgroups, etc. The higher the weight (higher priority) is, the lower the vruntime value. The process with the lowest vruntime is the next candidate for more CPU time.\nwait_sum Time spent waiting on a runqueue (in nanoseconds)\nwait_count Number of times waiting on a runqueue\nsum_exec_runtime – Total time process ran on CPU – In real time – Nano second units\n","permalink":"https://linux-audit.com/kernel/scheduler/explanation-of-values-in-proc-pid-sched/","tags":["kernel","linux","proc","scheduler","statistics"],"title":"Explanation of the values in /proc/PID/sched"},{"categories":["System administration"],"contents":"The renice tool is a command-line tool on Linux similar to command, except that it changes the priority of running processes. It can change the priority for multiple processes based on their process ID, process group ID, or even the user or users. This may help when a single process or multiple processes of a particular user are causing issues. Another option is that a particular task is taking too much time and needs more priority, where the system administrator then can decide to give that process more priority.\n","permalink":"https://linux-audit.com/system-administration/commands/renice/","tags":["linux","processes","scheduler","tools"],"title":"renice: change scheduler priority of a running process"},{"categories":["System administration"],"contents":"The nice tool can be used on Linux to start commands with a specified priority. This is useful to start tasks that may need to run for a while but at a low priority, to reduce the impact on the system or its performance.\n","permalink":"https://linux-audit.com/system-administration/commands/nice/","tags":["linux","processes","scheduler","tools"],"title":"nice: start a command with specified priority"},{"categories":["System administration"],"contents":"The chrt command can be used to change the policy and priority for the Linux scheduler that is applied to a running process or set it when running a new command. The command can also be used to display the current policy and priority.\n","permalink":"https://linux-audit.com/system-administration/commands/chrt/","tags":["linux","scheduler","tools"],"title":"chrt command"},{"categories":["System administration"],"contents":"The rev command can be used to read data and show the reversed version. When using it with files, it will reverse the output line by line. This tool is helpful with shell scripting when some data needs to be reversed.\n","permalink":"https://linux-audit.com/system-administration/commands/rev/","tags":["linux","data processing","tools"],"title":"rev command"},{"categories":["System administration"],"contents":"The blkid command can be used to show block device information on a Linux system. It shows similar information like /etc/fstab and specifics about the used block devices, such as block sizes.\n","permalink":"https://linux-audit.com/system-administration/commands/blkid/","tags":["linux","tools"],"title":"blkid command"},{"categories":["System administration"],"contents":"The units command provides a command-line tool to convert between different types of units. This way you can quickly convert between two units without having to search online.\nExamples include:\nCurrencies Energy Force Length Power Temperature Time ","permalink":"https://linux-audit.com/system-administration/commands/units/","tags":["linux","processes","troubleshooting"],"title":"units command"},{"categories":["System administration"],"contents":"The pslog command is a very small tool with one task: show which log files are opened by the selected process ID (PID). Just give it a PID and the program will show what log files are opened.\n","permalink":"https://linux-audit.com/system-administration/commands/pslog/","tags":["linux","processes","troubleshooting"],"title":"pslog command"},{"categories":["System administration"],"contents":"The prtstat command is a tool for Linux systems to show process information in one small overview. It retrieves its information from the /proc/PID/stat file and shows it in a formatted way.\nInformation that typically is included:\nProcess name Process state Number of threads User and group information Page faults CPU usage Memory usage Process scheduler details Example output:\nProcess: systemd-journal\tState: S (sleeping) CPU#: 0 TTY: 0:0\tThreads: 1 Process, Group and Session IDs Process ID: 549\tParent ID: 1 Group ID: 549\tSession ID: 549 T Group ID: -1 Page Faults This Process (minor major): 35655 135 Child Processes (minor major): 65 0 CPU Times This Process (user system guest blkio): 0,52 0,74 0,00 0,00 Child processes (user system guest): 0,00 0,00 0,00 Memory Vsize: 66 MB RSS: 20 MB RSS Limit: 18446744073709 MB Code Start: 0x63cedd9d7000\tCode Stop: 0x63cedd9f53c1 Stack Start: 0x7ffec869b400 Stack Pointer (ESP): 0\tInst Pointer (EIP): 0 Scheduling Policy: normal Nice: 0 RT Priority: 0 (non RT) ","permalink":"https://linux-audit.com/system-administration/commands/prtstat/","tags":["linux","processes","troubleshooting"],"title":"prtstat command"},{"categories":["System administration"],"contents":"The peekfd command is a tool for Linux systems to monitor file descriptors, usually open files and sockets. It shows any file descriptor activity for the process that is being tracked.\n","permalink":"https://linux-audit.com/system-administration/commands/peekfd/","tags":["linux","processes","troubleshooting"],"title":"peekfd command"},{"categories":["System administration"],"contents":"The pstree command is a tool for Linux systems to show active processes, including their hierarchy. When a process has child processes, it will be listed one level deeper. Due to the format it is called a tree.\nExample output:\nsystemd─┬─agetty ├─auditd───{auditd} ├─cron ├─dbus-daemon ├─dhclient ├─dovecot─┬─anvil │ ├─config │ └─log ├─nginx───2*[nginx] ├─qemu-ga───{qemu-ga} ├─sshd─┬─sshd───sshd───bash───su───bash │ └─sshd───sshd───bash───su───bash───pstree ├─systemd───(sd-pam) ├─systemd-journal ├─systemd-logind ├─systemd-timesyn───{systemd-timesyn} └─systemd-udevd ","permalink":"https://linux-audit.com/system-administration/commands/pstree/","tags":["linux","processes","troubleshooting"],"title":"pstree command"},{"categories":["System administration"],"contents":"The whatis tool is a search tool that retrieves information from the man pages. It returns a one-line description of any matches that were found in the locally available man pages. For more detailed searches, the apropos command can also be used.\n","permalink":"https://linux-audit.com/system-administration/commands/whatis/","tags":["linux","documentation","tools"],"title":"whatis: show one-line description for keyword"},{"categories":["System administration"],"contents":"The apropos tool is a search tool that retrieves information from the man pages that are installed on the system. It\u0026rsquo;s like a local search engine, matching related commands, system calls, and other system components. The command is easy to use, as it just needs a single word to search for to start doing its job. By using relevant command options the output can be reduced, such as filter on exact word matches or by using regular expressions.\n","permalink":"https://linux-audit.com/system-administration/commands/apropos/","tags":["linux","documentation","tools"],"title":"apropos: search keyword in available man pages"},{"categories":["Cheat sheets","System Administration"],"contents":"This cheat sheet helps performing tasks related to , such as sending DNS queries, lookup hostnames, and perform basic troubleshooting tasks.\nBasic usage Options Option Intended action -f FILE Perform multiple requests from a file (FILE) -t Specify record type -x Reverse DNS lookup (from IP to hostname) -4 Only use IPv4 -4 Only use IPv6 Query options Dig has specific query options, which start with a plus sign. They influence how output is presented. Query options usually also have a related query option, doing exactly the opposite. Depending on the query option, there is a default option. By having access to both, you can define the output that you like to see.\nQuery option Opposite Default option Intended action +all +noall +all Set or clear the display flags, will influence output +short +noshort +noshort Return only response or all information Basic request to look up a hostname The most simple method of using dig is just providing a hostname, such as a domain name or the .\n# dig linux-audit.com ; \u0026lt;\u0026lt;\u0026gt;\u0026gt; DiG 9.18.28-1~deb12u2-Debian \u0026lt;\u0026lt;\u0026gt;\u0026gt; linux-audit.com ;; global options: +cmd ;; Got answer: ;; -\u0026gt;\u0026gt;HEADER\u0026lt;\u0026lt;- opcode: QUERY, status: NOERROR, id: 37404 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;linux-audit.com.\tIN\tA ;; ANSWER SECTION: linux-audit.com.\t2280\tIN\tA\t89.41.171.41 ;; Query time: 0 msec ;; SERVER: 192.168.123.1#53(192.168.123.1) (UDP) ;; WHEN: Fri Dec 27 23:27:04 CET 2024 ;; MSG SIZE rcvd: 60 Change command output The output of dig is by default \u0026rsquo;noisy\u0026rsquo; and can be tuned. For example, provide \u0026#43;short to see just the output.\n# dig +short linux-audit.com 89.41.171.41 Want to see a little bit more?\ndig +noall +answer linux-audit.com linux-audit.com.\t4773\tIN\tA\t89.41.171.41 Note: order of the query options matter, especially when using \u0026lsquo;+noall\u0026rsquo;\nAnother interesting feature is displaying the output in the format.\n# dig +yaml linux-audit.com - type: MESSAGE message: type: RECURSIVE_RESPONSE query_time: !!timestamp 2024-12-27T23:09:21.576Z response_time: !!timestamp 2024-12-27T23:09:21.580Z message_size: 60b socket_family: INET socket_protocol: UDP response_address: \u0026#34;192.168.123.1\u0026#34; response_port: 53 query_address: \u0026#34;0.0.0.0\u0026#34; query_port: 0 response_message_data: opcode: QUERY status: NOERROR id: 34226 flags: qr rd ra QUESTION: 1 ANSWER: 1 AUTHORITY: 0 ADDITIONAL: 1 OPT_PSEUDOSECTION: EDNS: version: 0 flags: udp: 1232 QUESTION_SECTION: - \u0026#39;linux-audit.com. IN A\u0026#39; ANSWER_SECTION: - \u0026#39;linux-audit.com. 4949 IN A 89.41.171.41\u0026#39; Define DNS record type DNS uses different types of records to store information. Common record types include:\nA AAAA CNAME NS PTR SOA TXT We can query a specific record type by defining it, like querying which name servers are responsible for the google.com domain.\ndig +short NS google.com ns2.google.com. ns1.google.com. ns3.google.com. ns4.google.com. An alternative method is using the -t option, which does the same.\nWant to query multiple types?\ndig +noall +answer MX google.com NS google.com\nDefine DNS server For troubleshooting purposes it may be useful to use a different server than the one configured on your system. You can define which DNS resolver you want to use with the help of the @ sign.\n# dig @8.8.8.8 +short linux-audit.com 89.41.171.41 The dig command allows performing a set of queries starting at the root servers. Normally DNS resolving happens via a DNS resolver and that is responsible for starting at the beginning of the chain (the root servers) and follow each step, and finally return the related response to your system. With \u0026#43;trace you can let dig perform these steps, to see the responses.\ndig +trace google.com\n","permalink":"https://linux-audit.com/cheat-sheets/dig/","tags":["cheatsheet","command-line","dns","howto","linux","networking","one-liner","sockets","terminal"],"title":"dig cheat sheet"},{"categories":["System administration"],"contents":"The slabtop tool is similar to top but with focus on showing slab allocation. Slab allocation is a mechnism of the kernel to store memory pages as efficient as possible. Slabtop is useful for developers and system administrators to monitor or learn more about the memory usage and allocation of the Linux kernel.\n","permalink":"https://linux-audit.com/system-administration/commands/slabtop/","tags":["linux","memory","processes","tools"],"title":"slabtop: showing memory slab usage for the Linux kernel"},{"categories":["System administration"],"contents":"The basename tool is a helper tool for use in shell scripts to easily strip parts of a path, so that only the file name is returned. This simplifies automation tasks where you might need the core part of the file name and add a new extension to it, like when processing files and converting them.\n","permalink":"https://linux-audit.com/system-administration/commands/basename/","tags":["linux","memory","processes","tools"],"title":"basename: strip directory and file extension or suffix from path"},{"categories":["System Administration"],"contents":"Systemd comes with the systemctl command, which is the primary utility to query, introspect, or manage units. It can be used to show available units, including listing services. The basic subcommand list-unit-files can be combined with the --type option to select them.\nUsage root@debian-test:~# systemctl list-unit-files --type=service UNIT FILE STATE PRESET apache-htcacheclean.service disabled enabled apache-htcacheclean@.service disabled enabled apache2.service enabled enabled apache2@.service disabled enabled apparmor.service enabled enabled apt-daily-upgrade.service static - apt-daily.service static - auditd.service enabled enabled autovt@.service alias - console-getty.service disabled disabled console-setup.service enabled enabled container-getty@.service static - cron.service enabled enabled cryptdisks-early.service masked enabled cryptdisks.service masked enabled dbus-org.freedesktop.hostname1.service alias - dbus-org.freedesktop.locale1.service alias - dbus-org.freedesktop.login1.service alias - dbus-org.freedesktop.timedate1.service alias - dbus-org.freedesktop.timesync1.service alias - dbus.service static - debug-shell.service disabled disabled dovecot.service enabled enabled dpkg-db-backup.service static - e2scrub@.service static - e2scrub_all.service static - e2scrub_fail@.service static - e2scrub_reap.service enabled enabled emergency.service static - ... systemd-update-utmp.service static - systemd-user-sessions.service static - systemd-volatile-root.service static - udev.service alias - user-runtime-dir@.service static - user@.service static - x11-common.service masked enabled xpra.service disabled enabled 125 unit files listed. ","permalink":"https://linux-audit.com/systemd/faq/how-to-list-all-services-with-systemctl/","tags":["faq","linux","systemctl","systemd"],"title":"How to list all services with systemctl?"},{"categories":["System administration"],"contents":"The pidwait command helps with waiting for a task to complete. This is useful with shell scripting or tasks that may take a while to complete, but you don\u0026rsquo;t want to monitor the screen manually. While using pipes (command1 \u0026amp;\u0026amp; command 2) is possible to schedule the next task, pidwait will let you monitor the process state outside a piped set of commands. A good example of this is daemon processes that are running and normally are not started manually but with a script.\nTo use the command, the Linux kernel should be 5.3 or higher, as this command uses the syscall pidfd_open(2).\n","permalink":"https://linux-audit.com/system-administration/commands/pidwait/","tags":["linux","tools"],"title":"pidwait command"},{"categories":["System administration"],"contents":"The pmap tool retrieves the process mapping of a running process and displays this on screen with a defined level of formatting. When running the command without any specific options, it only requires the process ID (PID). To receive full output and see all details, most likely -XX provides the most level of detail.\nFields that pmap can show include:\nAddress Perm Offset Device Inode Size KernelPageSize* MMUPageSize Rss Pss Pss_Dirty Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous LazyFree AnonHugePages ShmemPmdMapped FilePmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked THPeligible ProtectionKey VmFlags Mapping ","permalink":"https://linux-audit.com/system-administration/commands/pmap/","tags":["linux","memory","processes","tools"],"title":"pmap command"},{"categories":["System administration"],"contents":"The pwdx tool is a very small tool with only one goal: show the current working directory of a process. It only required the process ID (PID) of a running process and then will show what the process is using as its value. This directory is used when no absolute path is given when performing some action to a path.\n","permalink":"https://linux-audit.com/system-administration/commands/pwdx/","tags":["linux","memory","processes","tools"],"title":"pwdx command"},{"categories":["System administration"],"contents":"The uptime command is a small utility on Linux that helps showing when the system was booted and how long it is running. Great for troubleshooting, system administration, and to learn more about the stability of a system.\n","permalink":"https://linux-audit.com/system-administration/commands/uptime/","tags":["linux","tools"],"title":"uptime command"},{"categories":["System administration"],"contents":"The watch command is one of those small Linux utilities that come in handy at times. It runs a command with a specified interval and shows its output. This is useful during troubleshooting, for example when you are interested in monitoring a directory for changes. Although you can define a \u0026lsquo;watch\u0026rsquo; with the Linux Audit Framework, it is quicker to run this command instead.\n","permalink":"https://linux-audit.com/system-administration/commands/watch/","tags":["linux","tools"],"title":"watch command"},{"categories":["System administration"],"contents":"The lynis command is a tool for Linux systems to perform a security audit of the system. It introspects the system configuration, package manager, and other components. The output is a set of warnings and suggestions to improve the security level of the system, which can help with system hardening.\n","permalink":"https://linux-audit.com/system-administration/commands/lynis/","tags":["audit","hardening","linux","linux security","security","server hardening","server security","system hardening","vulnerabilities","vulnerability scan","tools"],"title":"lynis command"},{"categories":["Nginx","Web"],"contents":"Nginx has so many options, that it is easy to see some never in practice. One of those is checking incoming HTTP requests for the presence of the Accept-Encoding header. This header defines what types of data encoding is supported. For modern clients this often means compressed data transfers with brotli, deflate, gzip, or zstd. In other words, the server can look what it supports and choose one of the algorithms to send back data compressed. This translates in shorter data transfers, saving bandwidth, and a smoother browsing experience.\nExample of header provided by the client:\nAccept-Encoding: gzip, deflate, br, zstd Block requests with missing Accept-Encoding header If we want to block all HTTP requests where is no support for data encoding, then we have to look at this Accept-Encoding header. If it is empty, then we can decide to block it.\nNext step is to return an error to the client. This is especially important for legitimate requests and clients, so that they know why they could not visit a page. Ironically, a modern browser does have all support for data encoding, so chances to see this are very small. Still, we should be good for other humans. For this reason we respond with HTTP code 406.\nThe HTTP 406 Not Acceptable client error response status code indicates that the server could not produce a response matching the list of acceptable values defined in the request\u0026rsquo;s proactive content negotiation headers and that the server was unwilling to supply a default representation.\nThis error code comes very close to why we won\u0026rsquo;t offer any data in return on the request.\nLet\u0026rsquo;s implement this within the nginx configuration. Usually this is within the context of a location:\nlocation / { # Block HTTP requests without data encoding support if ($http_accept_encoding = \u0026#39;\u0026#39;) { return 406 \u0026#34;Please configure your HTTP client to enable data compression (see Accept-Encoding header)\u0026#34;; } # ... other directives ... } After implementing the change, check your nginx configuration with nginx and the -t option:\nnginx -t\nThen restart the service, for Linux systems that is usually with systemctl.\nsystemctl restart nginx.service\nTesting if things work as expected, is easy using curl:\n# curl --head https://example.com/ HTTP/2 406 date: Wed, 25 Dec 2024 12:41:41 GMT content-type: application/octet-stream content-length: 90 strict-transport-security: max-age=31536000; includeSubDomains; preload Add the --compressed option to the command to test with data encoding enabled.\n# curl --compressed --head https://example.com/ HTTP/2 200 date: Wed, 25 Dec 2024 12:43:42 GMT content-type: text/html content-length: 7344 last-modified: Wed, 25 Dec 2024 00:11:44 GMT ...snipped output... ","permalink":"https://linux-audit.com/web/nginx-block-http-requests-without-data-compression-support/","tags":["brotli","gzip","howto","nginx"],"title":"Block HTTP requests for clients that don't offer data compression"},{"categories":["System administration"],"contents":"The fuser command is a tool for Linux systems to quickly identify which processes have a file or socket opened. This can be useful during unmounting a file system or finding our what processes have a connection active.\n","permalink":"https://linux-audit.com/system-administration/commands/fuser/","tags":["file system","linux","networking","sockets","troubleshooting"],"title":"fuser command"},{"categories":["System administration"],"contents":"The lsusb is a small utility for Linux to show what USB devices are connected to the system. When running it without any parameters, it will display what USB devices are available. With the options, a different output style can be achieved, including more verbose information about a particular device.\n","permalink":"https://linux-audit.com/system-administration/commands/lsusb/","tags":["binaries","hardware","kernel","linux","troubleshooting","usb"],"title":"lsusb command"},{"categories":["System administration"],"contents":"The setcap is a small utility to add or remove file capabilities and can be used together with the getcap command. Both may help better understanding the available Linux capabilities that are available to a binary and related process.\n","permalink":"https://linux-audit.com/system-administration/commands/setcap/","tags":["binaries","capabilities","kernel","linux","troubleshooting"],"title":"setcap command"},{"categories":["System administration"],"contents":"The getpcaps is a small utility to show process capabilities. This may help better understanding the available Linux capabilities that are available to a running process. While the tool has some parameters, for its basic purpose it just requires a PID .\n","permalink":"https://linux-audit.com/system-administration/commands/getpcaps/","tags":["binaries","capabilities","kernel","linux","troubleshooting"],"title":"getpcaps command"},{"categories":["System administration"],"contents":"The getcap is a very small utility that can show what file capabilities a provided binary has. This may help better understanding the available Linux capabilities that are available to the binary and related process.\n","permalink":"https://linux-audit.com/system-administration/commands/getcap/","tags":["binaries","capabilities","kernel","linux","troubleshooting"],"title":"getcap command"},{"categories":["System administration"],"contents":"The capsh is very helpful utility on Linux to learn more about Linux capabilities. It can display active capabilities and show information about them. The tool can also run other commands and show or define which Linux capabilities are applicable. This makes the tool useful for a wide range of tasks, such as troubleshooting, but also when you want to learn more about how processes are running. This information is useful when using containers, or securing Linux services with the systemd unit settings, such as CapabilityBoundingSet.\n","permalink":"https://linux-audit.com/system-administration/commands/capsh/","tags":["binaries","capsh","capabilities","kernel","linux","troubleshooting"],"title":"capsh command"},{"categories":null,"contents":"Why and when to use SecureBits Systemd provides the unit setting SecureBits to define a few special bits that are related to Linux capabilities. These bits is a flag named securebits and used with the operations PR_SET_SECUREBITS and PR_GET_SECUREBITS as part of the syscall prctl(2). The purpose of these bits is to alter the special treatment of user ID 0 (root) when it comes to capabilities.\nMapping of the systemd values and Linux securebits Keep capabilities The Linux securebits flag SECBIT_KEEP_CAPS allows a thread to retain its permitted capabilities, named the permitted set. Normally it would lose its permitted capabilities right away when switching from zero to a nonzero user ID. When this flag is set, and the effective UID is already nonzero, and the process thread switches all other user IDs to nonzero, then the effective capabilities remain. Otherwise the effective capabilities would be cleared when switching.\nThis flag is not compatible with the SECBIT_NO_SETUID_FIXUP flag, as it is a so-called superset and includes the SETBIT_KEEP_CAPS.\nSoftware may use older prctl(2) PR_SET_KEEPCAPS operation, which is the same as setting SECBIT_KEEP_CAPS.\nThe Linux kernel also has the securebits flag SECBIT_KEEP_CAPS_LOCKED. It makes SECBIT_KEEP_CAPS (keep-caps) immutable. So when this is set, no changes are allowed.\nRelated values for systemd SecureBits:\nkeep-caps keep-caps-locked Prevent setuid changes The securebits flag SECBIT_NO_SETUID_FIXUP prevents changing the kernel process\u0026rsquo;s permitted, effective, and ambient capability when set. It applies to the situation where the effective user and filesystem user ID is switched between zero and a nonzero value.\nTo lock this functionality (set immutable flag), the kernel has a SECBIT_NO_SETUID_FIXUP_LOCKED flag, which equals to the systemd SecureBits value of no-setuid-fixup-locked.\nRelated values for systemd SecureBits:\nno-setuid-fixup no-setuid-fixup-locked No root With the securebits flag SECBIT_NOROOT set, the kernel will not grant capabilities to a program when the setuid is set or the real UID of zero uses the syscall execve(2).\nThis flag can be also be made immutable using the SECBIT_NOROOT_LOCKED flag.\nRelated values for systemd for SecureBits:\nnoroot noroot-locked Generic advice Defining SecureBits and thereby changing the underlying securebits or prctl(2) flags definitely requires more knowledge of the software and service to be hardened.\nTesting With the systemd-run command we can test the individual values. To show the underlying securebits and their values, run capsh within the unit. Use a simple command like ps. Another interesting thing to test is what happens when sudo is involved.\n# systemd-run --pty --property=\u0026#34;SecureBits=noroot noroot-locked\u0026#34; capsh --print -- -c \u0026#34;sudo ps\u0026#34; Running as unit: run-p9196-i9496.service Press ^] three times within 1s to disconnect TTY. Current: = Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore Ambient set = Current IAB: Securebits: 03/0x3/2\u0026#39;b11 (no-new-privs=0) secure-noroot: yes (locked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=0(root) euid=0(root) gid=0(root) groups= Guessed mode: UNCERTAIN (0) sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted sudo: unable to open /etc/sudoers: Operation not permitted sudo: error initializing audit plugin sudoers_audit ","permalink":"https://linux-audit.com/systemd/settings/units/securebits/","tags":["capabilities","capsh","configuration","kernel","linux","ps","sandboxing","sudo","systemd","system hardening"],"title":"SecureBits setting"},{"categories":null,"contents":"Why and when to use RemoveIPC Systemd provides the unit setting RemoveIPC to perform a cleanup of IPC objects after a service exits. If this option is set, then upon the exit of a service all relevant IPC objects (System V and POSIX) will be removed.\nWhat is Inter-process communication (IPC)? Inter-process communication (IPC) provides processes with the capability to communicate with each other. This communication can happen via a variety of ways, using process signals, pipes, message queues, semaphores, and shared memory usage.\nSee What is inter-process communication (IPC)? for more details.\nGeneric advice For most common services this option can be set safely. If a process uses IPC, then most likely it will contain support for one of the common methods, such as message queues, semaphores or shared memory.\nSee filter set @ipc for syscalls that might be related.\n","permalink":"https://linux-audit.com/systemd/settings/units/removeipc/","tags":["configuration","hostname","ipc","linux","sandboxing","systemd","system hardening"],"title":"RemoveIPC setting"},{"categories":null,"contents":"Inter-process communication (IPC) provides processes with the capability to communicate with each other. This communication can happen via a variety of ways, using process signals, pipes, message queues, semaphores, and shared memory usage.\nSignals Processes can receive or send a signal to be informed about a specific event or action to take, such as stop running (SIGSTOP, SIGKILL, SIGTERM).\n# kill -l 1) SIGHUP\t2) SIGINT\t3) SIGQUIT\t4) SIGILL\t5) SIGTRAP 6) SIGABRT\t7) SIGBUS\t8) SIGFPE\t9) SIGKILL\t10) SIGUSR1 11) SIGSEGV\t12) SIGUSR2\t13) SIGPIPE\t14) SIGALRM\t15) SIGTERM 16) SIGSTKFLT\t17) SIGCHLD\t18) SIGCONT\t19) SIGSTOP\t20) SIGTSTP 21) SIGTTIN\t22) SIGTTOU\t23) SIGURG\t24) SIGXCPU\t25) SIGXFSZ 26) SIGVTALRM\t27) SIGPROF\t28) SIGWINCH\t29) SIGIO\t30) SIGPWR 31) SIGSYS\t34) SIGRTMIN\t35) SIGRTMIN+1\t36) SIGRTMIN+2\t37) SIGRTMIN+3 38) SIGRTMIN+4\t39) SIGRTMIN+5\t40) SIGRTMIN+6\t41) SIGRTMIN+7\t42) SIGRTMIN+8 43) SIGRTMIN+9\t44) SIGRTMIN+10\t45) SIGRTMIN+11\t46) SIGRTMIN+12\t47) SIGRTMIN+13 48) SIGRTMIN+14\t49) SIGRTMIN+15\t50) SIGRTMAX-14\t51) SIGRTMAX-13\t52) SIGRTMAX-12 53) SIGRTMAX-11\t54) SIGRTMAX-10\t55) SIGRTMAX-9\t56) SIGRTMAX-8\t57) SIGRTMAX-7 58) SIGRTMAX-6\t59) SIGRTMAX-5\t60) SIGRTMAX-4\t61) SIGRTMAX-3\t62) SIGRTMAX-2 63) SIGRTMAX-1\t64) SIGRTMAX\tPipes Pipes provide the output from one process to another one, typically for further processing.\nps -ef | grep systemd root 237 1 0 Dec17 ? 00:00:00 /lib/systemd/systemd-journald root 255 1 0 Dec17 ? 00:00:00 /lib/systemd/systemd-udevd systemd+ 311 1 0 Dec17 ? 00:00:00 /lib/systemd/systemd-timesyncd message+ 528 1 0 Dec17 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 533 1 0 Dec17 ? 00:00:00 /lib/systemd/systemd-logind michael 574 1 0 Dec17 ? 00:00:00 /lib/systemd/systemd --user root 12290 591 0 12:59 pts/0 00:00:00 grep systemd Semaphores Semaphores provided a location in memory of which the value can be view and test by multiple processes.\nRelated syscalls include:\nsemctl(2) semget(2) semop(2) semtimedop(2) Message queues A message queue provides two or more processes a way to communicate. Each of them opens the message queue using the mq_open(3) function call that includes a name of the queue. Messages then can be shared.\nRelated syscalls include:\nmq_getsetattr(2) mq_notify(2) mq_open(2) mq_timedreceive(2) mq_timedsend(2) mq_getsetattr(2) mq_timedreceive(2) mq_timedsend(2) mq_unlink(2) Shared memory By using shared memory, processes can communicate via shared virtual address space that is visible to them. Access is controlled via keys and by using some permission checks. When a process is allowed to use the shared memory, it can see those areas.\n","permalink":"https://linux-audit.com/what-is/ipc/","tags":["ipc","what-is"],"title":"What is inter-process communication (IPC)?"},{"categories":null,"contents":"Why and when to use PrivateUsers Systemd can make use of namespaces, including the user namespace. With the setting PrivateUsers, a service can be provided with a mapped set of user and group identities. This option, when enabled, will provide process capability isolation. This means the process capabilities within user name space does not have the same capabilities in the user namespace of the host itself. Within the namespace of the service it may have full capabilities, while on the namespace of the host it has none.\nConfiguration Using \u0026lsquo;self\u0026rsquo; or \u0026lsquo;yes\u0026rsquo; will do the following:\nMap root user and group Map user and group of unit (User=) Map every other user to user \u0026rsquo;nobody\u0026rsquo;, same for group This will introduce a split between the users within the service unit and the outside environment. Any file created by a user inside the service that is not known outside of it, will have \u0026rsquo;nobody\u0026rsquo; as its user and group. This option creates a sandboxed environment.\nWhen using \u0026lsquo;identity\u0026rsquo;, the first 65536 user and group IDs are mapped. Above that number, it will be mapped to the \u0026rsquo;nobody\u0026rsquo; user and group. While this is less strict than \u0026lsquo;self\u0026rsquo; (or yes), it will still provide process capability isolation.\nGeneric advice When possible, enable this setting to benefit from the process capability isolation.\n","permalink":"https://linux-audit.com/systemd/settings/units/privateusers/","tags":["authentication","configuration","linux","namespaces","processes","sandboxing","service hardening","systemd"],"title":"PrivateUsers setting"},{"categories":null,"contents":"Why and when to use KeyringMode Systemd has the unit setting KeyringMode that controls how the kernel session keyring is configured. It allows or prevents access to a keyring of a user. This way key material can be protected and blocked for processes that should not need access to it.\nGeneric advice For most system services, the \u0026lsquo;private\u0026rsquo; value is advised. This ensures that no user keyring is linked. This is especially useful for services running with the root user.\n[Service] KeyringMode=private To see if a program requires access to a keyring, track if the following syscalls are used.\nadd_key(2) request_key(2) ","permalink":"https://linux-audit.com/systemd/settings/units/keyringmode/","tags":["authentication","configuration","cryptography","linux","processes","sandboxing","service hardening","systemd"],"title":"KeyringMode setting"},{"categories":null,"contents":"Why and when to use ProtectHostname Systemd provides the unit setting ProtectHostname to restrict changing the hostname of the system. This makes use of the UTS namespace, which defines the hostname and NIS domain name. The hostname can normally be changed using the syscall sethostname(2), while the NIS domain name has the related setdomainname(2) syscall.\nSystemd leverages namespaces for isolation. It is done by creating a new UTS namespace that prevents making any changes to the host system. The creation of a new UTS namespace happens when calling clone(2) or unshare(2) syscall with the CLONE_NEWUTS flag. This functionality requires a Linux kernel that is compiled with the CONFIG_UTS_NS option.\nSyscalls of interest are:\nsethostname(2) setdomainname(2) If a program contains these system calls, then additional research is needed if ProtectHostname can be used safely.\nGeneric advice Setting the hostname is rarely needed, so most services can be configured with ProtectHostname=yes.\n","permalink":"https://linux-audit.com/systemd/settings/units/protecthostname/","tags":["configuration","hostname","linux","namespaces","sandboxing","systemd","system hardening"],"title":"ProtectHostname setting"},{"categories":null,"contents":"Why and when to use ReadOnlyPaths Systemd has the setting ReadOnlyPaths to grant read-only access. This might be needed when the service is hardened using the ProtectSystem that greatly reduces the access a service unit has to the file system. Another option is to use ReadOnlyPaths to mark the file system read-only, then use ReadWritePaths to open a few paths for writing.\nExample We can mark the file system as read-only to the service, except a few paths that we need to store our log or PID file.\n[Service] ReadOnlyPaths=/ ReadWritePaths=/run /var/log/application Another option is to protect the file system a bit further by not even allowing read access to most paths, except a few that we need.\n[Service] ProtectSystem=strict ReadOnlyPaths=/data/csv When a path is prefixed with a minus (-), it is ignored if it does not exist When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.g. configured with RootDirectory) Caveats This setting will not have effect if a process is missing the normal file permissions or ownership. For additional sandboxing, consider using \u0026lsquo;CapabilityBoundingSet=~CAP_SYS_ADMIN\u0026rsquo; or \u0026lsquo;SystemCallFilter=~@mount\u0026rsquo;.\nGeneric advice When possible, restrict file system access as much as possible by implementing ProtectSystem.\n","permalink":"https://linux-audit.com/systemd/settings/units/readonlypaths/","tags":["configuration","file system","linux","sandboxing","service hardening","systemd"],"title":"ReadOnlyPaths setting"},{"categories":null,"contents":"Why and when to use PrivateMounts Systemd has the unit setting PrivateMounts to provides the service with a private mount namespace. Only the service will see this view of the mount points and it will not be propagated to the other services on the same host. Mount points visible on the host will still be propagated to service though.\nGeneric advice This option can be useful to use, but typically is not needed if one or more of the following settings is already configured.\nBindPaths BindReadOnlyPaths InaccessiblePaths PrivateDevices PrivateTmp ProtectHome ProtectSystem ReadOnlyPaths ReadWritePaths ","permalink":"https://linux-audit.com/systemd/settings/units/privatemounts/","tags":["configuration","file system","linux","mount","service hardening","systemd"],"title":"PrivateMounts setting"},{"categories":null,"contents":"Introduction This is a hardening profile to help securing Dovecot, the POP3 and IMAP server by using a strict systemd unit configuration. The goal for this hardening profile is to sandbox the application while still allowing its tasks.\nRelevant FAQ: How to use systemctl edit to change a service?\nNotes ","permalink":"https://linux-audit.com/systemd/hardening-profiles/dovecot/","tags":["configuration","hardening","linux","software","systemd"],"title":"Dovecot hardening profile"},{"categories":null,"contents":"Why and when to use PrivateNetwork With the systemd unit setting PrivateNetwork a new network namespace can be defined. If enabled, then this means that all network interfaces will be invisible to the service. Only a local interface \u0026rsquo;lo\u0026rsquo; will be available, preventing any network communications with the host.\nGeneric advice For most services PrivateNetwork=yes is not an option when network communication is required. This option may however be useful for scripts that do not need any network access.\nCaveats Setting PrivateNetwork to yes will also set PrivateMounts to yes, unless PrivateMounts=no is specified. Before turning this on, investigate the impact of PrivateMounts.\nIf two services still need to communicate to each other, then this is possible using the JoinsNamespaceOf setting.\nTesting To see if a program works with this property, consider using the systemd-run command.\nWith the default boolean value of \u0026rsquo;no\u0026rsquo;, all links will be visible.\n# systemd-run --pty --property=PrivateNetwork=no ip link Running as unit: run-p3085-i3385.service Press ^] three times within 1s to disconnect TTY. 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp1s0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 52:52:52:bd:bd:bd brd ff:ff:ff:ff:ff:ff altname enx525252bdbdbd When set to boolean value \u0026lsquo;yes\u0026rsquo;, only the local interface will show up.\n# systemd-run --pty --property=PrivateNetwork=yes ip link Running as unit: run-p3079-i3379.service; invocation ID: fec3a19e27ba4226aeaba5a50a869b1e Press ^] three times within 1s to disconnect TTY. 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Trying to ping a local device on the network will fail.\nsystemd-run --pty --property=PrivateNetwork=yes ping -c 1 192.168.1.1 Running as unit: run-p3106-i3406.service Press ^] three times within 1s to disconnect TTY. /usr/bin/ping: connect: Network is unreachable ","permalink":"https://linux-audit.com/systemd/settings/units/privatenetwork/","tags":["configuration","linux","networking","sandboxing","service hardening","systemd"],"title":"PrivateNetwork setting"},{"categories":["Data processing"],"contents":"Linux systems have a wide range of useful utilities to do data processing, including searching through files. If you need to search for all unique words in a file, the grep command can perform this task very quickly. Let\u0026rsquo;s have a look at how to grep, followed by sorting and making them unique.\ngrep --only-matching --extended-regexp '[a-zA-Z]+' NEWS | sort | uniq\nExplanation We start the grep command with the --only-matching option to tell it to only show the results that match. Next step is to define what we are looking for. As we are not searching for a particular string, we use --extended-regexp to initiate a search with a regular expression. Next step is defining the regular expression ([a-zA-Z]+), meaning all words starting with a small or capital letter, with one or more occurrences.\nThe output of this file is a lot of words, including duplicates. To reduce the output, sorting can be done with sort, so uniq can filter all duplicates and make each line in the output unique.\n","permalink":"https://linux-audit.com/data-processing/faq/how-to-find-all-unique-words-in-a-file/","tags":["data","data processing","faq","grep","howto","linux","sort"],"title":"How to find all unique words in a file?"},{"categories":["System Administration"],"contents":"Systemd has multiple ways to show its version, support for features, paths, and user information. Let\u0026rsquo;s have a look at the different methods and what information it shows.\nUsing systemctl A quick way to check the systemd version and the features it supports is using systemctl with the --type option. The first line will show the version, the second includes what features are supported, which are typically compiled by a Linux distribution.\n# systemctl --version systemd 252 (252.31-1~deb12u1) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified Looking to retrieve just the version number?\n# systemctl --version | awk \u0026#39;{if($1==\u0026#34;systemd\u0026#34; \u0026amp;\u0026amp; $2~\u0026#34;^[0-9]\u0026#34;){print $2}}\u0026#39; | head -n 1 252 The command above queries the version and shows it if the first column contains the word \u0026lsquo;systemd\u0026rsquo; and the second column starts with a number. Only the first one line of output is returned, in case this output changes over time.\nStored in file: systemd.pc Systemd also stores information in the file /usr/share/pkgconfig/systemd.pc . It includes:\nPaths User ID ranges Systemd version # cat /usr/share/pkgconfig/systemd.pc # SPDX-License-Identifier: LGPL-2.1-or-later # # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # Names with prefixes are preferred, and the run-together names should be # considered deprecated (though there is no plan to remove them). New names # shall have underscores. prefix=/usr root_prefix= rootprefix=${root_prefix} sysconf_dir=/etc sysconfdir=${sysconf_dir} systemd_util_dir=${root_prefix}/lib/systemd systemdutildir=${systemd_util_dir} systemd_system_unit_dir=${rootprefix}/lib/systemd/system systemdsystemunitdir=${systemd_system_unit_dir} systemd_system_preset_dir=${rootprefix}/lib/systemd/system-preset systemdsystempresetdir=${systemd_system_preset_dir} systemd_user_unit_dir=${prefix}/lib/systemd/user systemduserunitdir=${systemd_user_unit_dir} systemd_user_preset_dir=${prefix}/lib/systemd/user-preset systemduserpresetdir=${systemd_user_preset_dir} systemd_system_conf_dir=${sysconfdir}/systemd/system systemdsystemconfdir=${systemd_system_conf_dir} systemd_user_conf_dir=${sysconfdir}/systemd/user systemduserconfdir=${systemd_user_conf_dir} systemd_system_unit_path=${systemd_system_conf_dir}:/etc/systemd/system:/run/systemd/system:/usr/local/lib/systemd/system:${systemd_system_unit_dir}:/usr/lib/systemd/system:/lib/systemd/system systemdsystemunitpath=${systemd_system_unit_path} systemd_user_unit_path=${systemd_user_conf_dir}:/etc/systemd/user:/run/systemd/user:/usr/local/lib/systemd/user:/usr/local/share/systemd/user:${systemd_user_unit_dir}:/usr/lib/systemd/user:/usr/share/systemd/user systemduserunitpath=${systemd_user_unit_path} systemd_system_generator_dir=${root_prefix}/lib/systemd/system-generators systemdsystemgeneratordir=${systemd_system_generator_dir} systemd_user_generator_dir=${prefix}/lib/systemd/user-generators systemdusergeneratordir=${systemd_user_generator_dir} systemd_system_generator_path=/run/systemd/system-generators:/etc/systemd/system-generators:/usr/local/lib/systemd/system-generators:${systemd_system_generator_dir} systemdsystemgeneratorpath=${systemd_system_generator_path} systemd_user_generator_path=/run/systemd/user-generators:/etc/systemd/user-generators:/usr/local/lib/systemd/user-generators:${systemd_user_generator_dir} systemdusergeneratorpath=${systemd_user_generator_path} systemd_sleep_dir=${root_prefix}/lib/systemd/system-sleep systemdsleepdir=${systemd_sleep_dir} systemd_shutdown_dir=${root_prefix}/lib/systemd/system-shutdown systemdshutdowndir=${systemd_shutdown_dir} tmpfiles_dir=${prefix}/lib/tmpfiles.d tmpfilesdir=${tmpfiles_dir} user_tmpfiles_dir=${prefix}/share/user-tmpfiles.d sysusers_dir=${prefix}/lib/sysusers.d sysusersdir=${sysusers_dir} sysctl_dir=${prefix}/lib/sysctl.d sysctldir=${sysctl_dir} binfmt_dir=${prefix}/lib/binfmt.d binfmtdir=${binfmt_dir} modules_load_dir=${prefix}/lib/modules-load.d modulesloaddir=${modules_load_dir} catalog_dir=${prefix}/lib/systemd/catalog catalogdir=${catalog_dir} system_uid_max=999 systemuidmax=${system_uid_max} system_gid_max=999 systemgidmax=${system_gid_max} dynamic_uid_min=61184 dynamicuidmin=${dynamic_uid_min} dynamic_uid_max=65519 dynamicuidmax=${dynamic_uid_max} container_uid_base_min=524288 containeruidbasemin=${container_uid_base_min} container_uid_base_max=1878982656 containeruidbasemax=${container_uid_base_max} Name: systemd Description: systemd System and Service Manager URL: https://systemd.io/ Version: 252 Using D-Bus Another option is using D-Bus and make a query on the message bus to retrieve information about systemd. This can be done using the busctl command.\n#busctl get-property org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager Version s \u0026#34;252.31-1~deb12u1\u0026#34; To query other properties, use the introspect command on the related message bus.\nbusctl introspect org.freedesktop.systemd1 /org/freedesktop/systemd1\nIt will show all methods, properties, and signals for the relevant nodes.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-systemd-version/","tags":["faq","linux","systemctl","systemd"],"title":"How to see the systemd version?"},{"categories":null,"contents":"Why and when to use PrivatePIDs Systemd can make use of namespaces, including the PID namespace. With the setting PrivatePIDs a service can be provided with such PID namespace. This way all processes within the namespace can\u0026rsquo;t see any processes running outside its own scope.\nGeneric advice This setting is only suitable for processes that do not fork themselves. The reason for this is that the first process will be assigned PID 1, the init process. The kernel is programmed to kill any processes if the init process stops. So this functionality should be used with services that have a type of simple or one-shot.\nIf the kernel does not support PID namespaces yet, it will be ignored.\n","permalink":"https://linux-audit.com/systemd/settings/units/privatepids/","tags":["configuration","linux","namespaces","processes","sandboxing","service hardening","systemd"],"title":"PrivatePIDs setting"},{"categories":["System Administration"],"contents":"Why harden systemd service units in the first place? Systemd service units are often configured by a basic set of settings. This allows most people to run the service without any issues. While that is fine, it also means that there is typically room for improvement, especially when it comes to security. Over the years many new unit settings were added, including some great systemd security features.\nHardening your own services is not difficult, but it requires a good approach to find the optimal balance between security and a running service. If you tighten the security measures a bit too much, then the service won\u0026rsquo;t work. If you are too sloppy, then you don\u0026rsquo;t benefit from the great sandboxing features that systemd has to offer. In this article we look at how to take a step-by-step approach, and increase the security measures in levels.\nHardening profiles With many people running the same software packages, we crafted some hardening profiles. These hardening profiles will give you a good foundation to start with. Depending on your situation and how a service is configured, a little bit of tuning might be needed. That is also why this article contains troubleshooting steps, so you can find the reason why a hardening profile might not always directly work for your system.\nThe following hardening profiles are currently available:\nApache Dovecot nginx OpenSMTPD This article is meant to support these hardening profiles and also shows how we came up with the settings.\nRestricting resources Systemd comes with two groups of settings that restrict resources. Usually they start with Restrict or Protect, but for some settings this might not be the case. Let\u0026rsquo;s have a look at common unit settings to further enhance the security posture of our system services.\nKeyringMode Implementation risk: (very low)\u0026#63; The kernel keyring provides key material to services, such as security data, encryption keys, and authentication information. If a service does not need access to the keyring of a particular user (including root), then systemd allows restricting this using the KeyringMode setting.\nWhen in doubt that key material is requested by a service, inspect the program code or use strace to track the following syscalls:\nadd_key(2) request_key(2) If no access is needed to key material, then lock access down.\n[Service] KeyringMode=private ProtectClock Implementation risk: (very low)\u0026#63; With the setting ProtectClock we can prevent a service from making any changes to the system clock.\nMost processes should only allowed to read clock information, but not modify it. The obvious exception to this is a service like a NTP daemon or program such as rdate. For most services it is therefore safe to prevent the service attempting to make changes to the system clock.\n[Service] ProtectClock=yes ProtectHostname Implementation risk: (very low)\u0026#63; A process rarely needs to change the hostname or NIS domain name of the system. In this case the ProtectHostname can be used to prevent this.\n[Service] ProtectHostname=yes To know if changes to the hostname or NIS domain name are needed, we can look for the following syscalls:\nsethostname(2) setdomainname(2) If these are not present, then this setting can be enabled.\nProtectKernelModules Implementation risk: (low)\u0026#63; Most services do not need to load new kernel modules. With systemd unit setting ProtectKernelModules the explicit loading of kernel modules can be blocked.\nThis setting can be applied to most system services. Some software, especially focused on network traffic capture, may use a custom kernel module and have the need to load it. But otherwise it is safe to block it, preventing any unauthorized loading of kernel modules.\n[Service] ProtectKernel=yes RestrictNamespaces Implementation risk: (low)\u0026#63; Software can leverage Linux namespaces to isolate system resources like a filesystem, user and group IDs, or even networking. If this is the case, then one of the following syscalls is used.\nclone(2) setns(2) unshare(2) Look in the source code for these syscalls, run strace or inspect a binary.\nstrings /lib/systemd/systemd-udevd | grep -E \u0026#34;(clone|setns|unshare)\u0026#34; setns Failed to clone sd_device object: %m RestrictRealtime Implementation risk: (low)\u0026#63; The Linux scheduler defines which process gets how much CPU resources, such as the time slice and at what priority. With the setting RestrictRealtime, systemd can block services from making use of several real-time policies.\nRelated syscalls to determine if a service needs access to real-time scheduling policies:\nsched_yield(2) sched_setaffinity(2) To find out if access may be needed, search in the source code, run strace, or inspect the related binary. For example for nginx, the output looks like this:\n# strings /usr/sbin/nginx | grep sched_ sched_setaffinity sched_yield sched_setaffinity(): using cpu #%ui sched_setaffinity() failed This section is under development and new settings are being added.\nRestricting executable paths Systemd service units typically will start a process, which then may execute others. By strongly defining what is allowed, we can harden our services to only execute those commands that are needed for its execution.\nRelated settings:\nExecPaths NoExecPaths The easiest way to find out what components are started or required is by using the Linux Audit Framework.\nStop service The first step is to stop the service, so we can do a clean start.\nsystemctl stop dovecot.service\nActivate audit rule and start service We are interested in all events where a binary is started. For Linux systems this means we are interested in the syscall execve(2).\nWith that in mind, we define our audit rule, where we capture the syscall with the -s option. The -k is used to label it with a key, which we later can use to quickly find the relevant entries.\nauditctl -a exit,always -F arch=b64 -S execve -k all-execve\nNote: this rule is defined on a 64-bits architecture, which is common, but may be different for your system.\nStart the service directly after enabling the audit rule, so the audit can log and we don\u0026rsquo;t have pollution from other processes.\nsystemctl start dovecot.service\nStop audit rule and search Let the software run for a bit, then disable the audit rules by deleting it.\nauditctl -D\nTime to query all entries that were happening upon activating the audit rule.\nausearch -i -t today -k all-execve\nThis will show a list of entries. As we are only interested in the lines mentioning the binaries on disk, we can filter a bit more.\n# ausearch -i -ts today -k all-execve | grep item=0 type=PATH msg=audit(12/16/2024 21:07:52.598:6323) : item=0 name=/usr/bin/systemctl inode=526685 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.602:6324) : item=0 name=/bin/systemd-tty-ask-password-agent inode=526697 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.634:6326) : item=0 name=/usr/sbin/dovecot inode=570398 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.638:6327) : item=0 name=/usr/bin/doveconf inode=570395 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.642:6328) : item=0 name=/usr/sbin/dovecot inode=570398 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.662:6330) : item=0 name=/usr/lib/dovecot/log inode=664678 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.666:6331) : item=0 name=/usr/lib/dovecot/anvil inode=664653 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:07:52.666:6332) : item=0 name=/usr/lib/dovecot/config inode=664656 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(12/16/2024 21:08:04.266:6334) : item=0 name=/usr/sbin/auditctl inode=570333 dev=fe:01 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 This list gives us a great start. For this list we exclude anything related to systemd itself (systemctl, systemd-tty-ask-password-agent) and the audit framework (auditctl).\nWith a little bit of scripting we can pull in the sixth field, sort, make it unique, strip out some commands, then show it as a single line:\n# ausearch -i -ts today -k all-execve | grep item=0 | awk \u0026#39;{print $6}\u0026#39; | awk -F= \u0026#39;{print $2}\u0026#39; | sort | uniq | grep -vE \u0026#34;(systemctl|systemd|auditctl)\u0026#34; | tr \u0026#39;\\n\u0026#39; \u0026#39; \u0026#39;` /usr/bin/doveconf /usr/lib/dovecot/anvil /usr/lib/dovecot/config /usr/lib/dovecot/log /usr/sbin/dovecot This line are the executables we at least need for our service to run. We can now define an explicit deny for the root path using NoExecPaths and add our allowed binaries to ExecPaths.\n[Service] NoExecPaths=/ ExecPaths=/usr/bin/doveconf /usr/lib/dovecot/anvil /usr/lib/dovecot/config /usr/lib/dovecot/log /usr/sbin/dovecot After adding these lines, it is time to restart the service and see if everything stays working.\nsystemctl restart dovecot.service\nRestricting capabilities and syscalls Most processes that run as a daemon will require some of the available Linux capabilities. Some developers define these capabilities clearly, but most of them don\u0026rsquo;t. In that case, we need to figure out what capabilities are required to operate correctly. As an extension to these capabilities, we have the syscalls that are used. These system functions allow the user space program to communicate with the kernel in a standardized way. To have a process working correctly, we need to make sure that it also can use the syscalls it requires, similarly to the capabilities. This is also where capabilities and syscalls come together, as usually the usage of syscalls give a very good hint on what capabilities are required.\nTo find about more about the capabilities and syscalls, we have a few options that we can use. Let\u0026rsquo;s have a look at them, so we can tune our systemd services the best way possible.\nIntrospection methods There are two ways to do some basic analysis of a service and the related process or processes. We start with strace, next is a systemd unit setting.\nOption 1: Using strace The strace command allows inspecting running processes. It usually requires root permissions and has a small risk of crashing a process, as the inspection is done using a few diversion paths. This also may impact the performance of the process. Best option to test is on a non-production system or on one where an accidental crash of a process it not a big issue. Although strace is often available, it may be missing, so use which strace to see if it is available.\nInspect and adjust the existing service The first action that we are going to take is to [edit a systemd unit] and add the strace command.\nFirst we want to find the current ExecStart value. We need this, so we can add it to our override file, prepended with the strace command.\nsystemctl cat dovecot.service | grep ExecStart\nNext step is to edit the service.\nsystemctl edit dovecot.service\nYour editor will open and it is time to define a [Service] block with two additional lines. The first one clears the existing ExecStart, while the second one add the strace command to it.\n[Service] ExecStart= ExecStart=/usr/bin/strace --absolute-timestamps=precision:us --daemonize --follow-forks --output=/tmp/strace.log /usr/sbin/dovecot -F Restart the service systemctl restart dovecot.service\nPerform some basic tasks The service should be running now, and strace will track what it is doing in the background. This may result in a lot of logging, so we just leave this on for a short moment of time, like a few minutes. In the case of a HTTP server, you could a few requests, for a mail server it would be useful to send an email, and so on.\nCopy the log file There should be a in log file, most likely stored as /tmp/systemd-private-IDENTIFIER-systemd-SERVICENAME-RANDOMSTRING/tmp/strace.log. Obviously the path is different on each system and run.\nStop the service and comment out the ExecStart lines Next step is to stop the service. Edit the service unit again and disable the lines by commenting it out. If the service needs to be running, start it again.\nFirst analysis of the strace log The log file will be filled with syscalls that are requested. While they might look cryptic at first, we can learn a lot about the functionality that a service needs. Let\u0026rsquo;s have a look at a few of those lines:\n43193 19:10:39.162977 execve(\u0026#34;/usr/sbin/dovecot\u0026#34;, [\u0026#34;/usr/sbin/dovecot\u0026#34;, \u0026#34;-F\u0026#34;], 0x7ffecf1f3558 /* 7 vars */) = 0 43193 19:10:39.163186 brk(NULL) = 0x560b8b386000 43193 19:10:39.163241 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2425f00000 43193 19:10:39.163264 access(\u0026#34;/etc/ld.so.preload\u0026#34;, R_OK) = 0 43193 19:10:39.163287 openat(AT_FDCWD, \u0026#34;/etc/ld.so.preload\u0026#34;, O_RDONLY|O_CLOEXEC) = 3 43193 19:10:39.163313 newfstatat(3, \u0026#34;\u0026#34;, {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0 43193 19:10:39.163338 close(3) = 0 43193 19:10:39.163358 openat(AT_FDCWD, \u0026#34;/usr/lib/dovecot/glibc-hwcaps/x86-64-v4/libsystemd.so.0\u0026#34;, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 43193 19:10:39.163417 newfstatat(AT_FDCWD, \u0026#34;/usr/lib/dovecot/glibc-hwcaps/x86-64-v4\u0026#34;, 0x7ffd1fb8bab0, 0) = -1 ENOENT (No such file or directory) 43193 19:10:39.163438 openat(AT_FDCWD, \u0026#34;/usr/lib/dovecot/glibc-hwcaps/x86-64-v3/libsystemd.so.0\u0026#34;, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) In this case we see execve(2), a call to execute a binary, followed by some file requests. At this moment we are not interested in the specifics yet. First action is to gather all the system calls. We can do this by getting the third column. Unfortunately, the third column is not just showing the system calls alone, but also some parameters. That is something we need to filter out.\nawk \u0026#39;{print $3}\u0026#39; strace-dovecot.log | awk -F\\( \u0026#39;{print $1}\u0026#39; | grep -E \u0026#34;^[a-z]\u0026#34; | sort | uniq -c | sort -k1 -n Breakdown of this command:\nPrint the third column using awk Split this output at the parentheses sign, we want to have the left part (syscall) To reduce some clutter, only show those names that start with lowercase character Sort the output Count all unique occurrences Sort the new output by the first key (the number of occurrences) and do this with numeric rules in mind The output is a list of matches and might look like this:\n1 capget 1 capset 1 exit_group 1 fdatasync 1 fstatfs 1 link 1 readlink 1 rt_sigreturn 1 sendmsg 1 socketpair 1 symlink 1 writev 2 chroot 2 mkdir 2 rename 2 rt_sigprocmask 2 sendto 2 sysinfo 2 wait4 3 chdir 3 dup 5 getpeername 5 uname 6 clone 6 mknodat 6 setgroups 7 getsockopt 7 getuid 8 alarm 8 setgid 8 setuid 9 arch_prctl 9 execve 9 geteuid 9 getgid 9 rseq 9 set_tid_address 10 epoll_create 10 getegid 11 munmap 14 chown 15 set_robust_list 16 prctl 19 futex 20 getdents64 23 getpid 25 prlimit64 26 getrandom 34 access 36 setsockopt 36 unlink 37 accept 39 listen 47 brk 52 connect 54 rt_sigaction 59 getsockname 63 bind 70 dup2 78 epoll_wait 81 write 86 lseek 87 pipe2 88 pread64 93 mprotect 99 umask 116 socket 253 epoll_ctl 298 read 323 close 381 mmap 452 openat 570 newfstatat 930 fcntl These syscalls are useful to look them up and see in which filter sets they belong to.\nOption 2: Using the SystemCallLog setting With the help of the systemd unit setting SystemCallLog we can log any any matches and is available since systemd 247. The interesting part of this setting is that we can tell it to only log those system calls that do or do NOT match.\nEnable the log setting Since most services need a basic set, we will be granting our service unit the @system-service filter set. So that is also the first set that we will define in the log. All syscalls that are NOT part of this set, can be discovered using the following configuration.\n[Service] SystemCallLog=~@system-service Restart service and check seccomp output If we restart our service and then filter on the recent items related to seccomp, we can find if anything would be blocked.\nsystemctl restart dovecot.service \u0026amp;\u0026amp; journalctl _AUDIT_TYPE_NAME=SECCOMP --since \u0026quot;1 min ago\u0026quot;\nWe see the following entry showing up:\nDec 16 23:46:47 debian-test audit[44560]: SECCOMP auid=4294967295 uid=0 gid=114 ses=4294967295 subj=unconfined pid=44560 comm=\u0026#34;anvil\u0026#34; exe=\u0026#34;/usr/lib/dovecot/anvil\u0026#34; sig=0 arch=c000003e syscall=161 compat=0 ip=0x7f7d7904db57 code=0x7ffc0000 In this case the syscall has number 161, which translates on x86_64 to the chroot(2) syscall. With help of the capabilities overview we can see that the chroot(2) syscall is part of the capability CAP_SYS_CHROOT. So we need to make sure that this program is able to properly use this functionality.\nWhen we look at the syscall filter sets used by systemd, then we can see that chroot is part of the filter set @privileged and @mount. The latter is a common filter set for system services. So besides giving the capability, we will grant the filter set @mount. Before we do that, we extend our existing logging.\nAdjusting the service [Service] SystemCallLog=~@mount @system-service We restart the service again, followed by the journalctl command. This time we only request the items of the very last minute.\n# systemctl restart dovecot.service \u0026amp;\u0026amp; journalctl _AUDIT_TYPE_NAME=SECCOMP --since \u0026#34;1 min ago\u0026#34; -- No entries -- No entries are displayed. Instead of @mount, we could also try if just chroot it enough. Let\u0026rsquo;s remove @mount and add chroot to the end of the list.\n[Service] SystemCallLog=~@system-service chroot Again it will show no entries, so we know that just granting chroot is already enough. This will restrict make the set as small as possible.\nEnable syscall filtering with SystemCallFilter Implementation risk: (high)\u0026#63; Now that we know what filter set(s) and syscall(s) we need, we can start enabling the syscall filtering with the SystemCallFilter setting.\nLet\u0026rsquo;s reorder the setting a bit and begin with the filter set(s), followed by the individual syscalls.\n[Service] SystemCallFilter=@system-service chroot Tip: you can rename Log into Filter, but don\u0026rsquo;t forget to remove the tilde (~) at the beginning of the line.\nRestart the service to test if everything is still working as expected.\nEnable capabilities filtering with CapabilityBoundingSet Capabilities are not as easily logged as syscalls. At the same time, we have seen the two options to gather the system calls. By looking them up in the capabilities overview we know that the chroot syscall requires CAP_SYS_CHROOT.\nA few common capabilities for processes to run:\nCapability Purpose Related syscalls CAP_CHOWN Allow changing file ownerships chown CAP_DAC_OVERRIDE Bypasses file read, write, and execute permission checks mount, utime, utimensat CAP_NET_BIND_SERVICE Bind a socket to a privileged port number below 1024 bind CAP_SETGID Allows making changes to the group ID of a process clone, getgroups, seteuid, setfsgid, setgid, setgroups, setresuid, setreuid CAP_SETUID Allows making changes to the user ID of a process clone, keyctl, seteuid, setfsuid, setresuid, setreuid, setuid So let\u0026rsquo;s add these items and allow chroot functionality.\n[Service] CapabilityBoundingSet=CAP_CHOWN CAP_DAC_OVERRIDE CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_CHROOT If all is well, a restart of the service should still succeed.\n","permalink":"https://linux-audit.com/systemd/how-to-harden-a-systemd-service-unit/","tags":["capabilities","linux","seccomp","scheduler","syscall","systemd","umask"],"title":"How to harden a systemd service unit"},{"categories":["System administration"],"contents":"Firejail is a program that can sandbox other programs. Its intended purpose is to restrict a program by limiting what a process and its children can see or do.\nTo achieve sandboxing functionality, Firejail uses Linux capabilities, namespaces, and secure computing in the form of seccomp-bpf. The current implementation of Firejail uses a SUID binary. That means that there is a risk of compromise if a bug in Firejail can be exploited. The project therefore recommends limiting who can access the tooling.\nAs Firejail is an alternative sandboxing tools to flatpak and snap, these will not work together.\nExamples Tracing DNS requests Firejail comes with a very handy option --dnstrace to see the initiated DNS requests. If a program is specified, then it will show the DNS request for the applicable namespaces. When providing just the option alone, all namespaces will be monitored. This is very useful to quickly see what DNS requests happen on a system.\n# firejail --dnstrace DNS trace for Sun Dec 15 00:02:21 2024 00:02:28 192.168.123.1 _http._tcp.security.debian.org (type 33) 00:02:28 192.168.123.1 _http._tcp.deb.debian.org (type 33) 00:02:28 192.168.123.1 debian.map.fastlydns.net (type 1) 00:02:28 192.168.123.1 debian.map.fastlydns.net (type 28) Blocking capabilities To see if a tool uses any capabilities, we can tell Firejail to block them all. If it is required for proper functioning, then it will show up very quickly.\n# firejail --caps.drop=all ping 192.168.1.1 /bin/bash: line 1: /usr/bin/ping: Operation not permitted So this needs filecap we can easily find that `ping` requires a capability. Which one? ```plaintext {hl_lines=1} # filecap /usr/bin/ping set file capabilities rootid effective /usr/bin/ping net_raw So in this case we need to allow the CAP_NET_RAW capability.\n# firejail --caps.keep=net_raw ping -c 1 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=0.474 ms --- 192.168.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.474/0.474/0.474/0.000 ms Now the ping command works again.\n","permalink":"https://linux-audit.com/system-administration/commands/firejail/","tags":["capabilities","kernel","linux","namespaces","sandboxing","seccomp","troubleshooting"],"title":"firejail command"},{"categories":["Network","System Administration"],"contents":"Almost any Linux system will be using DNS to translate between hostnames and IP addresses. To find out what DNS requests or queries are being made, there are multiple tools available. Learning about these requests is useful to better understand what incoming and outgoing connections happen or are required for proper functioning. It may help in securing your system or simply troubleshoot any issues.\nTools and examples Linux has many tools available that can help capturing network traffic, with even a few focused on the DNS itself. Let\u0026rsquo;s cover a few common options, including examples on its usage and related output.\ntcpdump A common utility to see network traffic is tcpdump. It has many available options available, making it is a useful tool for troubleshooting and monitoring purposes.\nTo use tcpdump, specify the network interface using the -i option. Not sure what interface? Use ip link and use that instead of enp1s0 in the example below. As we are interested in DNS requests, we define the port 53, so we capture both on UDP and TCP.\n# tcpdump -nnni enp1s0 port 53 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:16:22.372195 IP 192.168.123.100.48758 \u0026gt; 192.168.123.1.53: 48112+ SRV? _http._tcp.security.debian.org. (48) 12:16:22.372657 IP 192.168.123.100.33065 \u0026gt; 192.168.123.1.53: 17391+ SRV? _http._tcp.deb.debian.org. (43) 12:16:22.372773 IP 192.168.123.1.53 \u0026gt; 192.168.123.100.33065: 17391 1/0/0 SRV debian.map.fastlydns.net.:80 10 1 (87) 12:16:22.372878 IP 192.168.123.100.50772 \u0026gt; 192.168.123.1.53: 38773+ A? debian.map.fastlydns.net. (42) 12:16:22.372882 IP 192.168.123.100.50772 \u0026gt; 192.168.123.1.53: 34164+ AAAA? debian.map.fastlydns.net. (42) 12:16:22.372908 IP 192.168.123.1.53 \u0026gt; 192.168.123.100.50772: 38773 1/0/0 A 151.101.38.132 (58) 12:16:22.372916 IP 192.168.123.1.53 \u0026gt; 192.168.123.100.50772: 34164 1/0/0 AAAA 2a04:4e42:9::644 (70) 12:16:22.374429 IP 192.168.123.1.53 \u0026gt; 192.168.123.100.48758: 48112 1/0/0 SRV debian.map.fastlydns.net.:80 10 1 (92) 12:16:22.374518 IP 192.168.123.100.57846 \u0026gt; 192.168.123.1.53: 56403+ A? debian.map.fastlydns.net. (42) 12:16:22.374523 IP 192.168.123.100.57846 \u0026gt; 192.168.123.1.53: 42324+ AAAA? debian.map.fastlydns.net. (42) Firejail The firejail command is primarily intended for sandboxing, but can also be used to do some monitoring and inspecting. The output of the --dnstrace option is more friendly than that of tcpdump.\n# firejail --dnstrace DNS trace for Sun Dec 15 12:06:07 2024 12:06:13 192.168.123.1 google.com (type 1) 12:06:13 192.168.123.1 google.com (type 28) 12:06:13 192.168.123.1 110.39.251.142.in-addr.arpa (type 12) 12:06:14 192.168.123.1 110.39.251.142.in-addr.arpa (type 12) 12:06:16 192.168.123.1 _http._tcp.security.debian.org (type 33) 12:06:16 192.168.123.1 _http._tcp.deb.debian.org (type 33) 12:06:16 192.168.123.1 debian.map.fastlydns.net (type 28) 12:06:16 192.168.123.1 debian.map.fastlydns.net (type 1) dnstop One of the troubleshooting and monitoring tools focused on DNS is dnstop. It can provide various counters related to DNS requests, such as:\nRequests by source or destination IP Requests by TLD Requests by domain name Query types Response codes # dnstop enp1s0 -l 3 Queries: 0 new, 6 total Query Name Count % cum% ------------------- --------- ------ ------ map.fastlydns.net 4 66.7 66.7 deb.debian.org 1 16.7 83.3 security.debian.org 1 16.7 100.0 dnscap An alternative tool to capture DNS requests is the dnscap utility. It also has a wide range of options available and requires at least the -g option to start the capture.\n# dnscap -g [76] 2024-12-15 13:58:45.235064 [#0 enp1s0 4095] \\ [192.168.123.100].49956 [192.168.123.1].53 \\ dns QUERY,NOERROR,10127,rd \\ 1 _http._tcp.security.debian.org.,IN,SRV 0 0 0 [71] 2024-12-15 13:58:45.235501 [#1 enp1s0 4095] \\ [192.168.123.100].46064 [192.168.123.1].53 \\ dns QUERY,NOERROR,27576,rd \\ 1 _http._tcp.deb.debian.org.,IN,SRV 0 0 0 [115] 2024-12-15 13:58:45.235632 [#2 enp1s0 4095] \\ [192.168.123.1].53 [192.168.123.100].46064 \\ dns QUERY,NOERROR,27576,qr|rd|ra \\ 1 _http._tcp.deb.debian.org.,IN,SRV \\ 1 _http._tcp.deb.debian.org.,IN,SRV,112,[32] 0 0 [70] 2024-12-15 13:58:45.235726 [#3 enp1s0 4095] \\ [192.168.123.100].56351 [192.168.123.1].53 \\ dns QUERY,NOERROR,27217,rd \\ 1 debian.map.fastlydns.net.,IN,A 0 0 0 [70] 2024-12-15 13:58:45.235730 [#4 enp1s0 4095] \\ [192.168.123.100].56351 [192.168.123.1].53 \\ dns QUERY,NOERROR,27986,rd \\ 1 debian.map.fastlydns.net.,IN,AAAA 0 0 0 [86] 2024-12-15 13:58:45.235746 [#5 enp1s0 4095] \\ [192.168.123.1].53 [192.168.123.100].56351 \\ dns QUERY,NOERROR,27217,qr|rd|ra \\ 1 debian.map.fastlydns.net.,IN,A \\ 1 debian.map.fastlydns.net.,IN,A,112,151.101.38.132 0 0 [98] 2024-12-15 13:58:45.235761 [#6 enp1s0 4095] \\ [192.168.123.1].53 [192.168.123.100].56351 \\ dns QUERY,NOERROR,27986,qr|rd|ra \\ 1 debian.map.fastlydns.net.,IN,AAAA \\ 1 debian.map.fastlydns.net.,IN,AAAA,112,2a04:4e42:9::644 0 0 ","permalink":"https://linux-audit.com/networking/faq/how-to-see-all-dns-requests-on-the-system/","tags":["dns","faq","howto","linux","monitoring","network","tcpdump","troubleshooting"],"title":"How to see all DNS requests on the system?"},{"categories":null,"contents":"Introduction This is a hardening profile to help securing the Apache web server by using a strict systemd unit configuration. The goal for this hardening profile is to limit what Apache can do, while still being able to serve HTTP requests.\nRelevant FAQ: How to use systemctl edit to change a service?\nNotes Depending on external components such as PHP, temporary files may need to be made accessible. When possible, define an alternative directory and keep PrivateTmp enabled.\nPossible errors Apache no longer starts after applying the profile Job for apache2.service failed because the control process exited with error code. See \u0026#34;systemctl status apache2.service\u0026#34; and \u0026#34;journalctl -xeu apache2.service\u0026#34; for details. This may happen due to a profile that is too strict for your system or environment. Perform the troubleshooting steps for failed systemd units to see what needs to be changed.\nThe journal shows errors Depending on the Linux distribution, Apache might be started with a wrapper script. Specific binaries like /usr/bin/id and /usr/bin/rm might be needed.\nDec 15 22:20:44 test apachectl[37748]: /usr/sbin/apachectl: 155: rm: Permission denied ","permalink":"https://linux-audit.com/systemd/hardening-profiles/apache/","tags":["apache","configuration","hardening","linux","systemd"],"title":"Apache hardening profile"},{"categories":["System administration"],"contents":"The netcap utility shows the current capabilities for processes that are using network sockets.\nWhen running the netcap command it will retrieve the active processes that have network sockets opened. For each of those processes, it will show what capabilities they have assigned. If they are unrestricted, in other words all capabilities, it will show full in the capabilities column.\nSpecial characters are the @, meaning ambient capabilities, and + to show open-ended bounding set, meaning it is not restricted.\nThis tool is very helpful to quicly focus on network-related applications and see what capabilities they have. This may help in system hardening efforts, for example by using a hardening profile to systemd services.\nExample output Example of running netcap as the root user.\n# netcap ppid pid acct command type port capabilities 1 20728 root nginx tcp 80 full + 1 515 root sshd tcp 22 full + 515 552 root sshd tcp 22 full + 515 15485 root sshd tcp 22 full + 1 20728 root nginx tcp6 80 full + 1 515 root sshd tcp6 22 full + 1 403 root dhclient udp 68 dac_override, net_bind_service, net_admin, net_raw + 1 403 root dhclient pkt enp1s0 dac_override, net_bind_service, net_admin, net_raw + ","permalink":"https://linux-audit.com/system-administration/commands/netcap/","tags":["capabilities","command","kernel","linux","network"],"title":"netcap command"},{"categories":["System administration"],"contents":"The pscap utility shows the current capabilities that a process has access to. When running the pscap command it will retrieve the active processes and determines from each process what capabilities they have assigned. If they are unrestricted, in other words all capabilities, it will show full in the capabilities column.\nSpecial characters are the @, meaning ambient capabilities, and + to show open-ended bounding set, meaning it is not restricted.\nThis tool is very helpful to quickly focus on network-related applications and see what capabilities they have. This may help in system hardening efforts, for example by using a hardening profile to systemd services\nExample output Example of running pscap as the root user.\n# pscap ppid pid name command capabilities 1 239 root systemd-journal chown, dac_override, dac_read_search, fowner, setgid, setuid, sys_ptrace, sys_admin, audit_control, mac_override, syslog, audit_read + 1 268 root systemd-udevd chown, dac_override, dac_read_search, fowner, fsetid, kill, setgid, setuid, setpcap, linux_immutable, net_bind_service, net_broadcast, net_admin, net_raw, ipc_lock, ipc_owner, sys_module, sys_rawio, sys_chroot, sys_ptrace, sys_pacct, sys_admin, sys_boot, sys_nice, sys_resource, sys_tty_config, mknod, lease, audit_write, audit_control, setfcap, mac_override, mac_admin, syslog, block_suspend, audit_read, perfmon, bpf, checkpoint_restore + 1 306 systemd-timesync systemd-timesyn sys_time @ + 1 403 root dhclient dac_override, net_bind_service, net_admin, net_raw + 1 502 root cron full + 1 504 messagebus dbus-daemon audit_write + 1 506 root qemu-ga full + 1 508 root systemd-logind chown, dac_override, dac_read_search, fowner, linux_immutable, sys_admin, sys_tty_config, audit_control, mac_admin + 1 510 root login full + 1 515 root sshd full + 1 541 root systemd full + 541 542 root (sd-pam) full + 510 548 root bash full + 515 552 root sshd full + 567 571 root su full + 571 572 root bash full + 515 15485 root sshd full + 15492 15496 root su full + 15496 15497 root bash full + 1 20728 root nginx full + ","permalink":"https://linux-audit.com/system-administration/commands/pscap/","tags":["capabilities","command","linux","kernel","processes"],"title":"pscap command"},{"categories":["System administration"],"contents":"The captest utility shows the current capabilities that a process has access to. It can be used for troubleshooting, testing, or demonstrating purposes.\nCaptest performs a set of of tasks and tests:\nShow current capabilities Try to access /etc/shadow and show the result Create a child process Try to access the shadow file again, including the result Show what capabilities the child has This functionality may be useful to learn more about capabilities, test containers, but also if a hardened systemd service is working correctly.\nExample output As an unprivileged user:\n$ captest User credentials uid:1000 euid:1000 suid:1000 Group credentials gid:1000 egid:1000 sgid:1000 Current capabilities: none securebits flags: none Attempting direct access to shadow...FAILED (Permission denied) Attempting to access shadow by child process...FAILED Child User credentials uid:1000 euid:1000 suid:1000 Child Group credentials gid:1000 egid:1000 sgid:1000 Child capabilities: none Child securebits flags: none Do not allow child process to regain privileges.\n# captest --lock User credentials uid:0 euid:0 suid:0 Group credentials gid:0 egid:0 sgid:0 Effective: 000001FF, FFFFFFFF Permitted: 000001FF, FFFFFFFF Inheritable: 00000000, 00000000 Bounding Set: 000001FF, FFFFFFFF Ambient : 00000000, 00000000 securebits flags: NOROOT, NOROOT_LOCKED, NO_SETUID_FIXUP, NO_SETUID_FIXUP_LOCKED Attempting direct access to shadow...SUCCESS Attempting to access shadow by child process...SUCCESS Child User credentials uid:0 euid:0 suid:0 Child Group credentials gid:0 egid:0 sgid:0 Child capabilities: none Child securebits flags: NOROOT, NOROOT_LOCKED, NO_SETUID_FIXUP, NO_SETUID_FIXUP_LOCKED ","permalink":"https://linux-audit.com/system-administration/commands/captest/","tags":["capabilities","kernel","linux","privilege escalation","troubleshooting"],"title":"captest command"},{"categories":["System administration"],"contents":"The filecap utility shows the current capabilities from the binaries defined in the PATH or specified directory.\nThis tool is useful to easily see which binaries currently have capabilities set on them via file system flags.\nExample output Scan $PATH for binaries and their capabilities.\n# filecap set file capabilities rootid effective /usr/bin/ping net_raw Filecap does not work if the binary path specified is a symbolic link to another directory.\n# filecap /bin Must be one regular file or directory Show all available capabilities. See the capabilities overview for more details about them.\n# filecap -d chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease audit_write audit_control setfcap mac_override mac_admin syslog wake_alarm block_suspend audit_read perfmon bpf checkpoint_restore ","permalink":"https://linux-audit.com/system-administration/commands/filecap/","tags":["binaries","capabilities","command","kernel","linux"],"title":"filecap command"},{"categories":null,"contents":"Why and when to use IPAccounting Systemd unit setting IPAccounting can be used to turn on accounting. If set, it will track the number of network packets and bytes that was send to the related service. It will both incoming (ingress) and outgoing (egress).\nConfiguration The default option is that IP accounting is disabled for services. This can be validated by looking at the relevant counters, such as IPIngressBytes and IPEgressBytes. The first one will track how much incoming traffic there was (in bytes), the second for outgoing traffic.\n# systemctl show nginx.service | grep -E \u0026#34;^IP\u0026#34; IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] If there is already information available, then IP accounting was enabled in the service or by setting DefaultIPAccounting=yes in a global systemd configuration file.\nExample to activate IPAccounting If you want to monitor your nginx service to see how much traffic is coming in, then we need to make adjustments to the service unit.\nOverride the service unit and add IPAccounting=yes under [Service].\nRestart nginx to activate the new settings.\nsystemctl restart nginx.service\nWhen traffic goes to or comes from the nginx service, it will become visible in the relevant counters.\n# systemctl show nginx.service | grep -E \u0026#34;^IP\u0026#34; IPIngressBytes=5908 IPIngressPackets=24 IPEgressBytes=3325 IPEgressPackets=13 Generic advice This setting can be helpful to monitor the traffic that goes to a specific service. Especially if it is unclear how much a service is being accessed.\n","permalink":"https://linux-audit.com/systemd/settings/units/ipaccounting/","tags":["accounting","configuration","linux","networking","syscall","systemd"],"title":"IPAccounting setting"},{"categories":null,"contents":"Introduction This is a hardening profile to help securing OpenSMTPD by using systemd unit configuration.\nRelevant FAQ: How to use systemctl edit to change a service?\nNotes This is the first version and currently under testing. Want to help get it to work, share feedback or issues, let it know!\n","permalink":"https://linux-audit.com/systemd/hardening-profiles/opensmtpd/","tags":["configuration","hardening","linux","email","systemd"],"title":"OpenSMTPD hardening profile"},{"categories":["System Administration"],"contents":"This cheat sheet is focused on the pacman command, the package manager for systems like Arch Linux.\nBasic usage Pacman uses mostly command options and their action might not always be clear, especially when using the shortened aliases. So here is a basic list with the options in long and short format, followed by the intended action.\nCommand Short version Intended action --query --deps --unrequired -Qdt Find all packages that are installed as dependency but no longer required --remove PACKAGE -R Remove specified package --remove --recursive PACKAGE -Rs Remove package and dependencies, except those linked to other packages or manually installed --sync PACKAGE -S Install a package --sync --search REGEX -Ss Search for a package --sync --refresh -Sy Refresh the package database information Search a package If you know the name or a part of the name, then we can combine --sync with --search to search with a regular expression.\n# pacman -Ss lynis extra/lynis 3.1.2-1 Security and system auditing tool to harden Unix/Linux systems Example using a regular expression.\npacman -Ss \u0026quot;lyn*\u0026quot;\nInstallation, update, upgrade, and removal of packages Install new package Install a single package.\npacman -S lynis\nUpdate the database information pacman -Sy\nRequires root permissions\nUpgrading packages To update and upgrade the full system.\npacman -Syu\nRemove a package Remove a single package\npacman -R lynis\nRemove a package including its dependencies, except those dependencies that are linked to another installed package or were manually installed by the user.\npacman -Rs lynis\nRemove a package and also its dependencies.\npacman -Rsc lynis\nFile information Synchronize the package database and search for files with a particular name.\n# pacman -Fy lynis :: Synchronizing package databases... core 1110,9 KiB 7,38 MiB/s 00:00 [-----------------------] 100% extra 44,3 MiB 84,1 MiB/s 00:01 [-----------------------] 100% multilib 223,7 KiB 1912 KiB/s 00:00 [-----------------------] 100% extra/lynis 3.1.2-1 usr/bin/lynis usr/share/bash-completion/completions/lynis To search with a regular expression, provide also the --regex option.\nsudo pacman -Fxy lyni :: Synchronizing package databases... endeavouros is up to date core is up to date extra is up to date multilib is up to date extra/lynis 3.1.2-1 usr/bin/lynis usr/lib/systemd/system/lynis.service usr/lib/systemd/system/lynis.timer usr/share/bash-completion/completions/lynis usr/share/man/man8/lynis.8.gz extra/tela-circle-icon-theme-black 2024_11_15-2 usr/share/icons/Tela-circle-black/scalable/apps/dev.lynith.Speedtest.svg extra/tela-circle-icon-theme-blue 2024_11_15-2 usr/share/icons/Tela-circle-blue/scalable/apps/dev.lynith.Speedtest.svg extra/tela-circle-icon-theme-brown 2024_11_15-2 usr/share/icons/Tela-circle-brown/scalable/apps/dev.lynith.Speedtest.svg extra/tela-circle-icon-theme-dracula 2024_11_15-2 usr/share/icons/Tela-circle-dracula/scalable/apps/dev.lynith.Speedtest.svg Maintenance Packages that are no longer required Search for any dependencies that are no longer required.\npacman -Qdt\n","permalink":"https://linux-audit.com/cheat-sheets/pacman/","tags":["cheatsheet","howto","linux","package manager","pacman"],"title":"pacman cheat sheet"},{"categories":["System Administration"],"contents":"Systemd service units can be restarted with systemctl. There are three subcommands available to restart a service:\nreload restart reload-or-restart try-reload-or-restart Reload: systemctl reload Some services support reloading its configuration, typically without the need to fully restart. For example, a web server service like nginx does support this. In this case, a simple reload is enough.\nsystemctl reload nginx.service\nNot sure if a reload is possible? With subcommand show we can query the relevant property ExecReload from a service.\n# systemctl show --property ExecReload --value nginx.service { path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[Mon 2024-12-09 04:31:45 EST] ; stop_time=[Mon 2024-12-09 04:31:45 EST] ; pid=4050588 ; code=exited ; status=0 } Restart: systemctl restart If reloading is not enough, or the service does not support it, then a restart is required. This will effectively stop and start the service unit.\nsystemctl restart nginx.service\nReload or restart A hybrid option is to first reload the service. If that is not supported, then the service will be instructed to do a normal restart. This is achieved using the reload-or-restart subcommand.\nsystemctl reload-or-restart nginx.service\nTry a reload or restart For scripting purposes, the try-reload-or-restart subcommand will attempt to take the reload and restart action. If the service is not running, then no action will be performed.\nsystemctl try-reload-or-restart nginx.service\n","permalink":"https://linux-audit.com/systemd/faq/how-to-reload-or-restart-a-systemd-service/","tags":["faq","howto","kill","linux","systemd"],"title":"How to reload or restart a systemd service?"},{"categories":["System Administration"],"contents":"Systemd units can be enabled, making them available for startup during boot time of the system.\nTo see if a unit, such as a service, is actually enabled, use the is-enabled subcommand of systemctl.\nUsing systemctl is-enabled Using the subcommand is very easy, as it only requires a unit name.\n# systemctl is-enabled ssh.service enabled In this case the service is showing enabled, meaning it is enabled. An alternative way is to look at the output of the status subcommand. In that case look at the line starting with Loaded: and look at the value directly after the unit name.\n# systemctl status ssh.service ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-10-14 17:29:58 UTC; 1 month 24 days ago Docs: man:sshd(8) man:sshd_config(5) Main PID: 403504 (sshd) Tasks: 1 (limit: 2219) Memory: 488.0K CPU: 21ms CGroup: /system.slice/ssh.service └─403504 \u0026#34;sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\u0026#34; Notice: journal has been rotated since unit was started, output may be incomplete. ","permalink":"https://linux-audit.com/systemd/faq/how-to-check-if-a-systemd-service-is-enabled/","tags":["faq","howto","linux","systemd"],"title":"How to check if a systemd service is enabled?"},{"categories":["System Administration"],"contents":"Systemd units can use environment variables and forward them to an application. To set an enviroment variable, use the Environment or EnvironmentFile option.\nDefine Environment The basic option to set an environment variable is using Environment followed by the variable name and value.\n[Service] Environment=\u0026#34;MYVAR=hello world\u0026#34; This defines the variable MYVAR with the value \u0026lsquo;hello world\u0026rsquo;\nDefine EnvironmentFile Environment variables can also be stored within a file by using the option EnvironmentFile. This option might be very useful for automatic deployments, especially if a service unit is used within different environments (such as DEV and PROD). The systemd unit can remain the same, while the file containing the variables needs to be filled with the correct variables and values.\n[Service] EnvironmentFile=/data/env/MYAPP When using a file to store the environment variables, use the format \u0026ldquo;VAR=VALUE\u0026rdquo; (without quotes)\n","permalink":"https://linux-audit.com/systemd/faq/how-to-set-environment-variables-in-a-systemd-unit/","tags":["faq","howto","linux","systemd"],"title":"How to set environment variables in a systemd unit?"},{"categories":["System Administration"],"contents":"Depending on your Linux system, the /etc/resolv.conf file might be a symbolic link pointing to a so-called stub of systemd that handles DNS requests via systemd-resolved. Otherwise, it is a regular file and contains the DNS servers and some optional configuration settings, such as time-outs.\n","permalink":"https://linux-audit.com/system-administration/files/etc-resolv-conf/","tags":["linux"],"title":"/etc/resolv.conf: nameserver configuration"},{"categories":["System Administration"],"contents":"Usually /dev/random is created during the boot cycle using the mknod command. The random numbers provided are generated from an entropy pool that is filled using noise such as from device drivers. The underlying random functions also track the number of bits in the entropy pool, and may give a warning if it is not sufficient.\n","permalink":"https://linux-audit.com/system-administration/files/dev-random/","tags":["linux","random"],"title":"/dev/random: providing random numbers"},{"categories":["Authentication","Passwords"],"contents":"Password security strength typically comes from the complexity, including length, of the password itself. There is an additional security measure that can be implemented on Linux systems: hashing rounds. It does not strengthen the password itself, but improves the authentication phase. Let\u0026rsquo;s have a look in more detail.\nWhat are hashing rounds? A hashing algorithm computes if the result or outcome of a provided password is the same as a stored value. Not the actual password is checked, but the result of pulling it through a mathematical function is compared. If more computing resources are required (work factor), we increase the burden on an attacker. Within some hashing functions, this is called hashing rounds.\nMore rounds will increase the processing time, including the time it takes to authenticate. A normal user will have to endure a small delay in processing once when logging in, while the attacker encounters it for every single password cracking attempt. Some of the hashing methods have a greater resistance to so-called offline attacks. These attacks occur to a password database and is performed on a resource of the attack itself.\nDetermining current hashing algorithm Before we make any changes, it is good to know what hashing algorithm is used. This can be done by looking at the /etc/shadow file. The file is split with a colon (:) and we are interested in the second field. This is the field after the username. This second field also has columns, but this time is the separator a dollar sign ($).\nWe can query the shadow information using the getent command and show a particular user:\n# getent shadow test1 test1:$y$j9T$mYr.K6XvvNzs2gFziibUD1$BlrpAYrh5t0MFjbe.W8WJO2gWzkb9V6kWuGIu74tBg7:20062:0:99999:7::: To simplify this output a little bit, we can use awk to only extract the entries with a password and column to turn it into a nice table.\n# getent shadow | awk -F: \u0026#39;$2 ~ /^\\$/\u0026#39; | column --table --separator :$ test1 y j9T mYr.K6XvvNzs2gFziibUD1 BlrpAYrh5t0MFjbe.W8WJO2gWzkb9V6kWuGIu74tBg7 20062 0 99999 7 test2 y j9T mcWzdGP7ACjHzLonVuGvO/ AYFX0qOItF0cbcSR2ay9MAXj0uUlED1GVHkp9hMu2zA 20062 0 99999 7 test3 y j9T uf65A1fRAvZqAT7cVItuf. yQa8j9qpVnQKMhnu9x8Rnnu0Q8r1v0VfMu.MIgnvMHB 20062 0 99999 7 In this case we see the value \u0026lsquo;y\u0026rsquo; in the second column. The \u0026lsquo;y\u0026rsquo; refers to yescrypt. To see the full list of methods, have a look at password security with the /etc/shadow file for details.\nConfiguration using /etc/login.defs SHA-256 and SHA-512 Requires ENCRYPT_METHOD set to SHA256 or SHA512 in /etc/login.defs.\nDefine the setting SHA_CRYPT_MIN_ROUNDS. By default this setting is 5000. For modern systems, higher values are advised.\nThe number of rounds can also be defined in PAM. When using both, ensure that both are having a similar configuration.\nYescrypt Requires ENCRYPT_METHOD set to YESCRYPT in /etc/login.defs.\nWith yescrypt, there is a cost factor involved, which is similar to rounds. If your Linux distribution supports this, you will find the option YESCRYPT_COST_FACTOR in /etc/login.defs\nTo confirm that the change is working, update a password of a user. Then have a look at the third column in the output above. The value j9T refers to default of 5 rounds. Value jAT=6 rounds, jBT=7 rounds, jCT=8 rounds, jDT=9 rounds, jET=10 rounds, jFT=11 rounds.\nNot all Linux distributions support YESCRYPT_COST_FACTOR at this moment, so in that case try the \u0026lsquo;rounds\u0026rsquo; option in PAM.\nConfiguration PAM Linux typically has PAM available. Common file locations:\n/etc/pam.d/common-password /etc/pam.d/system-auth Look for the line starting with password and uses the pam_unix.so module.\nExample line:\npassword\t[success=1 default=ignore]\tpam_unix.so obscure yescrypt rounds=8 This line needs to be extended with \u0026lsquo;rounds=NUMBER\u0026rsquo;, which is equal to the cost factor of 8.\nIn this example we set round to 8 (valid options: 1-11)\n","permalink":"https://linux-audit.com/authentication/linux-password-security-hashing-rounds/","tags":["linux","password"],"title":"Linux password security: hashing rounds"},{"categories":["Software"],"contents":"The risks of having a compiler installed A compiler is a toolkit to turn source code into executable code, for Linux often ELF . The compiler might be used by malicious software to create a binary, backdoor, or even a Linux kernel module. Sometimes the attacker needs to compile the code to match the system and the right kernel. By removing the compiler, you make it harder to abuse the compiler to build a malicious piece of binary code.\nWhile a determined person might have alternative ways to get a compiler onto the system, it will take additional time and effort. Another possibility is that the attacker can only write to parts of the file system that is not executable, making it even harder. For that reason, it is also a good idea to shield the basic binary paths where possible. For example, do not allow PHP software to execute system commands if that is not needed for basic functionality.\nDoes my system get really much more secure when removing compilers? Just removing the compiler does not make the system a magnitude more secure. When combined with other security measures, it can be seen as a layered approach to security. Each layer adding a bit more security, making the chance of compromise smaller.\nBenefits of removing compilers Attack surface One of the security measures one can take is limiting installed software and active services. Everything that is not installed or running, can\u0026rsquo;t be misused. In security terms this is called reducing the attack surface of a system. By removing any compilers that are not needed for the functioning of the system, the attack surface of the system is reduced.\nLess overhead Another benefit of removing any unneeded software application, including compilers, is a lower footprint of the system. A smaller number of packages on the system means also less space used on the disk. Any backup of the disk will be smaller and therefore quicker to search or restore when needed. A package that is not installed, does also not require any updating, saving in time and bandwidth.\nWhat can be done if you still need a compiler? Sometimes you simply need a compiler available, as it is required for things like virtual machine guest tools. If that is the case, then one thing to consider is limiting the access to the root user or admin group. Remove the world readable bit from the binary.\nchmod 700 /usr/bin/gcc\n","permalink":"https://linux-audit.com/software/why-remove-compilers-from-your-system/","tags":["backdoor","compiler","packages"],"title":"Why remove compilers from your system?"},{"categories":["System Administration"],"contents":"The logs from OpenSSH are useful for monitoring and taking security measures. The OpenSSH daemon (sshd) typically uses the /var/log/auth.log file to store any information regarding relevant events such authentication attempts. Some systems may no longer use this file, for example in the case that systemd is being used. In that case, use the journalctl command to view the relevant entries from the journal logs.\nIf /var/log/auth.log is present Many Linux distributions have the auth.log file available in the /var/log directory. This file can be reviewed using common utilities:\ncat grep less more tail Another option is to open it in your preferred editor, although that might not be advisable due to the size the log file can become.\nTo track any changes to the file, use the tail command.\ntail -f /var/log/auth.log\nTo search a particular user in the log, grep can be used.\ngrep USERNAME /var/log/auth.log\nSystemd: use journalctl If your are running a newer distribution version, your /var/log/auth.log file might be missing. In that case check out the journal logs.\njournalctl -u ssh.service\n","permalink":"https://linux-audit.com/ssh/faq/how-to-see-the-ssh-log/","tags":["faq","howto","linux","logging","log files","openssh","ssh"],"title":"How to see the SSH log?"},{"categories":["Software","System Administration"],"contents":"Run apt command with the download subcommand to only download the package without actually installing it. This function may be useful when you want to add a package to your own software repository, store a backup of a package, or want to retrieve some information from the package archive file itself.\nExample of downloading a package with apt apt download lynis\n","permalink":"https://linux-audit.com/software/package-manager/faq/how-to-only-download-a-package-with-apt/","tags":["apt","faq","howto","linux","package manager","software"],"title":"How to download a package with apt without installing it?"},{"categories":null,"contents":"Why and when to use RuntimeDirectoryMode The setting defines the octal mode for the directories that are created for \u0026lsquo;runtime\u0026rsquo;. The default value is 0755, making it possible for other processes to access the underlying directory. Depending on the type of unit, the directory below the primary path will get assigned the related permissions.\nApplicable to Path System units /run Users units $XDG_RUNTIME_DIR The related environment variable is named $RUNTIME_DIRECTORY and contains the related location.\nCurrent configuration Look in the service itself to see the RuntimeDirectory property, as this contains the runtime directory.\nUse systemctl with the cat subcommand to retrieve the information more easily, including any existing unit overrides.\n# systemctl cat systemd-timesyncd.service | grep RuntimeDirectory RuntimeDirectory=systemd/timesync Another option is to review the settings by querying the properties of a service using systemctl with show, followed the service. To limit the information to just the property RuntimeDirectory, only that line will be returned.\n# systemctl show --property=RuntimeDirectory systemd-timesyncd.service RuntimeDirectory=systemd/timesync This will reveal not the full path, but as this is a system unit, it is prepended with the /run directory.\n# stat /run/systemd/timesync File: /run/systemd/timesync Size: 60 Blocks: 0 IO Block: 4096 directory Device: 1ah/26d\tInode: 1734 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 102/systemd-timesync) Gid: ( 104/systemd-timesync) Access: 2024-09-03 09:51:57.097000157 +0200 Modify: 2024-09-03 09:52:27.119040190 +0200 Change: 2024-09-03 09:52:27.119040190 +0200 Birth: 2024-09-03 09:51:57.097000157 +0200 This service has the default file permissions (0755).\nChanging RuntimeDirectoryMode property To change the file permissions for the runtime directory, edit the service and define the new file permissions (e.g. 0700).\nsystemctl edit myservice.service\nThen add the property to the override file, including the section Service if that is not present yet.\n[Service] RuntimeDirectoryMode=0700 Generic advice This option can be applied to most services that use a runtime directory. To find all possible service units that can be tuned, query all services and select the relevant property. To make it easier to see to what service a directory belongs to, select both the Id as RuntimeDirectory property.\nsystemctl show --type=service --property=Id,RuntimeDirectory '*'\n","permalink":"https://linux-audit.com/systemd/settings/units/runtimedirectorymode/","tags":["configuration","file permissions","linux","sandboxing","service hardening","systemd"],"title":"RuntimeDirectoryMode setting"},{"categories":null,"contents":"The run0 command was introduced in systemd version 256 and is intended as an alternative to sudo. Both commands elevate privileges, but are slightly different. According to systemd author Lennart Poettering, run0 as somewhat more similar to the ssh command than it is to sudo.\nTo learn how to use run0, have a look at the run0 introduction or at the cheat sheet.\n","permalink":"https://linux-audit.com/what-is/run0/","tags":["systemd","what-is"],"title":"What is run0?"},{"categories":["System Administration"],"contents":"Before reloading the SSH configuration, perform a configuration test first. This may avoid locking yourself out of a remote system or break any existing communications between the server and any clients. Testing can be done using the sshd command. For this purpose, it has the -t option available.\nCommand line usage To use this option on the command line, run the following command:\nsshd -t\n","permalink":"https://linux-audit.com/ssh/faq/how-to-test-sshd-configuration-for-errors/","tags":["configuration","faq","howto","linux","openssh","ssh","sshd"],"title":"How to test the sshd configuration for configuration errors?"},{"categories":null,"contents":"SSH is short for secure shell and is a network protocol to provide secure remote access to systems. It comes with strong authentication and encrypted data communication. SSH is used on Linux, BSD, and other Unix-based operating systems, with SSH clients available to other operating systems.\nSSH details Name Value Default port 22 Default protocol TCP More information about SSH Learn more about SSH: SSH configuration and settings. Topics include:\nSSH agent SSH client configuration SSH server configuration SSH keys ","permalink":"https://linux-audit.com/what-is/ssh/","tags":["ssh","what-is"],"title":"What is SSH?"},{"categories":["Software","System Administration"],"contents":"Run apt command with the show subcommand to see basic information about the package. This includes version information, but also any dependencies, recommendations, or even conflicting packages.\n# apt show systemd Package: systemd Version: 252.30-1~deb12u2 Priority: important Section: admin Maintainer: Debian systemd Maintainers \u0026lt;pkg-systemd-maintainers@lists.alioth.debian.org\u0026gt; Installed-Size: 9895 kB Provides: systemd-sysusers (= 252.30-1~deb12u2), systemd-tmpfiles (= 252.30-1~deb12u2) Pre-Depends: libblkid1 (\u0026gt;= 2.24), libc6 (\u0026gt;= 2.34), libcap2 (\u0026gt;= 1:2.10), libgcrypt20 (\u0026gt;= 1.10.0), liblz4-1 (\u0026gt;= 0.0~r122), liblzma5 (\u0026gt;= 5.1.1alpha+20120614), libmount1 (\u0026gt;= 2.30), libselinux1 (\u0026gt;= 3.1~), libssl3 (\u0026gt;= 3.0.0), libzstd1 (\u0026gt;= 1.5.2) Depends: libacl1 (\u0026gt;= 2.2.23), libaudit1 (\u0026gt;= 1:2.2.1), libblkid1 (\u0026gt;= 2.24.2), libcryptsetup12 (\u0026gt;= 2:2.4), libfdisk1 (\u0026gt;= 2.33), libkmod2 (\u0026gt;= 15), libp11-kit0 (\u0026gt;= 0.23.18.1), libseccomp2 (\u0026gt;= 2.3.1), libsystemd-shared (= 252.30-1~deb12u2), libsystemd0 (= 252.30-1~deb12u2), mount Recommends: default-dbus-system-bus | dbus-system-bus, systemd-timesyncd | time-daemon Suggests: systemd-container, systemd-homed, systemd-userdbd, systemd-boot, systemd-resolved, libfido2-1, libqrencode4, libtss2-esys-3.0.2-0, libtss2-mu0, libtss2-rc0, polkitd | policykit-1 Conflicts: consolekit, libpam-ck-connector, systemd-shim Breaks: less (\u0026lt;\u0026lt; 563), resolvconf (\u0026lt;\u0026lt; 1.83~), sicherboot (\u0026lt;\u0026lt; 0.1.6), udev (\u0026lt;\u0026lt; 247~) Homepage: https://www.freedesktop.org/wiki/Software/systemd Tag: admin::boot, implemented-in::c, interface::daemon, role::program, works-with::software:running Download-Size: 3035 kB APT-Manual-Installed: yes APT-Sources: http://deb.debian.org/debian bookworm/main amd64 Packages Description: system and service manager systemd is a system and service manager for Linux. It provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. . Installing the systemd package will not switch your init system unless you boot with init=/lib/systemd/systemd or install systemd-sysv in addition. To see the dependencies of a package that is not installed yet, use the install subcommand. It will show any additional packages that will be installed, which are the required dependencies of the selected package.\n# apt install nginx Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: nginx-common Suggested packages: fcgiwrap nginx-doc ssl-cert The following NEW packages will be installed: nginx nginx-common 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 640 kB of archives. After this operation, 1696 kB of additional disk space will be used. Do you want to continue? [Y/n] ","permalink":"https://linux-audit.com/software/package-manager/faq/how-to-see-dependencies-of-a-package-with-apt/","tags":["apt","faq","howto","linux","package manager","software"],"title":"How to see the dependencies of a package with apt?"},{"categories":["Software","System Administration"],"contents":"The opposite of installing a package is using apt with the remove subcommand. It will remove any defined file or directory that is available in the package listing. Any changes to files on this list will be left alone, so that you won\u0026rsquo;t lose them. This is useful when reinstalling a package. Same applies for directories that are not empty or that contain files added later, for example by the system administrator.\napt remove PACKAGE\nRun apt command with the purge subcommand to remove all files and do a full clean of a package and its previously installed files.\napt purge PACKAGE\n","permalink":"https://linux-audit.com/software/package-manager/faq/how-to-remove-a-package-with-apt/","tags":["apt","faq","howto","linux","package manager","software"],"title":"How to remove a package with apt?"},{"categories":["Software","System Administration"],"contents":"After packages are upgraded on systems running apt, the package manager have mark some packages as no longer required. Typically this happens when a dependency on another package has been removed, or another package supersedes the existing package. In that case the package becomes an orphan and no longer provides specific function another package or to the system.\nRun the apt command with the autoremove subcommand to check for orphaned packages and remove them.\napt autoremove\n","permalink":"https://linux-audit.com/software/package-manager/faq/how-to-remove-unused-packages-with-apt/","tags":["apt","faq","howto","linux","package manager","software"],"title":"How to remove unused packages with apt?"},{"categories":["System Administration"],"contents":"Systemd has a set of utilities available to monitor the state of units. One of the systemd commands is systemd-analyze that may help with troubleshooting issues, such as errors in unit files.\nTesting with systemd-analyze Run the systemd-analyze command with the verify subcommand, followed by the unit itself.\nNo errors # systemd-analyze verify ssh.service When there are no issues, the output remains empty.\nAssignment outside of section /etc/systemd/system/apparmor.service.d/override.conf:1: Assignment outside of section. Ignoring. This error indicates that a change has been made, but not inside a section, such as the \u0026lsquo;[Service]\u0026rsquo; block.\nUnknown key /etc/systemd/system/apparmor.service.d/override.conf:2: Unknown key \u0026#39;Befored\u0026#39; in section [Service], ignoring. When an unknown or invalid setting has been specified, systemd-analyze will show this as a unknown key.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-verify-a-systemd-unit-for-errors/","tags":["faq","linux","systemctl","systemd","systemd-analyze","troubleshooting"],"title":"How to verify a systemd unit for errors?"},{"categories":["System Administration"],"contents":"Systemd may show units, like services, as not-found in the LOAD column. Typically this is caused that these units are not available to systemd. There are a few causes for this to happen, such as being listed as a dependency for another unit. Typically units with the \u0026rsquo;not-found\u0026rsquo; state are not having an issue, especially if they are just a soft target for other units.\nExample output When filtering the output of the list-units subcommand on services, we can include the --all option to show them all.\n# systemctl list-units --type=service --all UNIT LOAD ACTIVE SUB DESCRIPTION apparmor.service loaded active exited Load AppArmor profiles apport-autoreport.service loaded inactive dead Process error reports when automatic reporting is enabled apport.service loaded active exited LSB: automatic crash report generation apt-daily-upgrade.service loaded inactive dead Daily apt upgrade and clean activities apt-daily.service loaded inactive dead Daily apt download activities ● auditd.service not-found inactive dead auditd.service auth-rpcgss-module.service loaded inactive dead Kernel Module supporting RPCSEC_GSS In this case it is the auditd.service that could not be found. The reason is simple, auditd is not installed on the system. So why does systemd, and systemctl in particular, list it in the first place? That is because it is a target defined in the sshd.service.\nAfter=network.target auditd.service ","permalink":"https://linux-audit.com/systemd/faq/why-does-systemctl-list-units-show-units-as-not-found/","tags":["faq","linux","systemctl","systemd"],"title":"Why does systemctl list-units show units as 'not-found'?"},{"categories":["System administration"],"contents":"Monotonic timers are timers that do not rely on the system clock, but at an arbitrary point. Such a point is typically an event, like a service becoming active, or the operating system completed the boot process. The timer can not move backwards, as it only gets triggered upon some system actions. The system clock therefore has no impact on the timer itself.\nOn Linux we see monotonic timers being used in systemd. For example, a timer could be configured with the OnBootSec setting.\nLearn more about systemd timers on Linux and how to configure them.\n","permalink":"https://linux-audit.com/what-is/monotonic-timer/","tags":["clock","what-is"],"title":"What is a monotonic timer?"},{"categories":["System Administration"],"contents":"The systemctl command can be used to show the state of units, including timers. For timers, there is a specific subcommand list-timers available.\nThe information that is presented, includes:\nNext run of the timer How much time is left before this next run occurs Last time the timer was active Time passed since this last active run Timer unit name Related service unit that will be triggered Usage # systemctl list-timers NEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2024-11-21 10:58:24 CET 43min left Wed 2024-11-20 10:58:24 CET 23h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Thu 2024-11-21 18:01:44 CET 7h left Thu 2024-11-21 08:05:28 CET 2h 9min ago apt-daily.timer apt-daily.service Fri 2024-11-22 00:00:00 CET 13h left Thu 2024-11-21 00:00:24 CET 10h ago dpkg-db-backup.timer dpkg-db-backup.service Fri 2024-11-22 00:00:00 CET 13h left Thu 2024-11-21 00:00:24 CET 10h ago logrotate.timer logrotate.service Fri 2024-11-22 04:37:08 CET 18h left Thu 2024-11-21 06:42:28 CET 3h 32min ago man-db.timer man-db.service Fri 2024-11-22 06:10:48 CET 19h left Thu 2024-11-21 06:48:24 CET 3h 26min ago apt-daily-upgrade.timer apt-daily-upgrade.service Sun 2024-11-24 03:10:56 CET 2 days left Sun 2024-11-17 03:10:24 CET 4 days ago e2scrub_all.timer e2scrub_all.service Mon 2024-11-25 01:13:09 CET 3 days left Mon 2024-11-18 01:12:54 CET 3 days ago fstrim.timer fstrim.service 8 timers listed. Pass --all to see loaded but inactive timers, too. ","permalink":"https://linux-audit.com/systemd/faq/how-to-see-active-systemd-timers/","tags":["faq","linux","systemctl","systemd"],"title":"How to see active systemd timers"},{"categories":["File Systems","System Administration"],"contents":"In the introduction to file permissions we learned about file permissions on Linux. When creating a new file, how does the system know what setting to use? It depends on different aspects.\nFile creation permissions When creating files, Linux looks at a few factors that influence the final outcome on the ownership and individual file permissions.\nuser that created the file context in the creation command is running specific parameters of the creation command umask value So it depends on the user that creates a file, but also the underlying command that is used. One way to create a file is by redirecting some text to a file, using the built-in shell redirection.\necho \u0026quot;test\u0026quot; \u0026gt; /tmp/mytestfile\nIn this case, the test string will be added to a file. If it does not exist yet, it will be created. The umask value within the shell defines in that case what the file permissions should be. We will have a look at the configuration and values.\nAnother option is using a dedicated command like touch to create the file. Running processes may also create new files, for example to a log file that does not exist yet. Typically when a command or process is creating a new file, it will leverage a syscall such as open(2).\nUmask values explained When seeing a umask for the first time, like 022, it does not directly look normal file permissions. That is because the value is inversed, as the umask was written with the idea what file permission bits to turn off.\nTo found out what a value like 022 means, we need to take each number. Then we can find out the values for directories by substraction the value from 7, or 6 for files.\nA file created with a umask of 022 will be:\n6-0 = 6 6-2 = 4 6-2 = 4 So that equals to 644 when using the chmod command.\nFor a directory this same umask will result in the same as chmod 755 directory.\nTip: with umask -S you can see the symbolic representation\nConfiguration places of umask The umask value can be configured in several files, that may configure it system-wide or per user.\n/etc/passwd /etc/pam.d/common-session /etc/profile /etc/login.defs /etc/default/login shell configuration file systemd unit file passwd file To set the umask for an individual user, the /etc/passwd file can be used. In the fifth field within the so-called GECOS information field, the umask details can be added.\nusername:x:1000:1000:User,,,,umask=0027:/home/username:/bin/bash\nAn easier way to do this with a command is using the chfn command.\nchfn -o \u0026quot;umask=0027\u0026quot; username\n/etc/profile The file /etc/profile is a system-wide .profile used by most common shells. It can be used to set a umask value.\nCaveat: this setting won\u0026rsquo;t always be used as it requires an active shell. For example, a graphical session or SSH session may therefore miss the umask value.\nPAM A common place to configure the umask is within the PAM configuration and specifically in the file /etc/pam.d/common-session . It requires the pam_umask module, which may need to be installed first.\nsession optional pam_umask.so umask=027 When using systemd, the UMask unit setting may be used for a service or a user service.\n[Service] UMask=0027 Also some applications allow defining the umask value within its own configuration. Obviously, the umask value may therefore differ for one user depending on the tools or applications being used.\n","permalink":"https://linux-audit.com/filesystems/file-permissions/set-default-file-permissions-with-umask/","tags":["file permissions","file system","linux","umask"],"title":"Set default file permissions on Linux with umask"},{"categories":null,"contents":"Why and when to use UMask Systemd provides the unit setting UMask to define the default umask. This value is used when creating new files or directories and defines the file permissions. It is an extension to the umask value that is normally already configured within the shell itself.\nConfiguration Set the value of UMask to a similar value as the umask setting in the shell, for example 022 or the full format 0022.\nGeneric advice This option can be applied to most services. It is useful to tighten up file permissions for those services that create new files, especially if the more generic umask settings are less strict.\nTesting To test if the umask correctly works, the systemd-run command can be used.\nCreate file that is only readable by the owner Run the command and set UMask to the value 0277. Then let the touch command create a new file.\nsystemd-run --pty --property=UMask=0277 touch /tmp/testfile\n","permalink":"https://linux-audit.com/systemd/settings/units/umask/","tags":["configuration","linux","sandboxing","service hardening","systemd","umask"],"title":"UMask setting"},{"categories":null,"contents":"Why and when to use RestrictRealtime Systemd provides the setting RestrictRealtime which aims to restrict the usage of real-time scheduling by a service.\nWhen using this option enabled, a service won\u0026rsquo;t be able to use realtime scheduling policies of the Linux scheduler, such as:\nSCHED_FIFO SCHED_RR SCHED_DEADLINE Relevant syscalls include:\nsched_setaffinity(2) sched_yield(2) Generic advice This option can be applied to many services. Processes focusing on performance or require a fair bit of CPU power (more than average), may leverage the scheduler functionality from the Linux kernel.\nTo find out if any scheduling policy is used, considering looking in the code of the program for the related syscalls or use the strings command.\nstrings /usr/sbin/nginx | grep sched_\n","permalink":"https://linux-audit.com/systemd/settings/units/restrictrealtime/","tags":["configuration","linux","sandboxing","scheduler","service hardening","systemd"],"title":"RestrictRealtime setting"},{"categories":null,"contents":"Why and when to use RestrictSUIDSGID The setting RestrictSUIDSGID aims to restrict the usage of the set-user-ID and set-group-ID bit on binaries.\nWhen using this option enabled, a service won\u0026rsquo;t be able to set the bits on a file.\nGeneric advice This option can be applied to almost all services. It is rarely needed to allow a service setting these bits.\n","permalink":"https://linux-audit.com/systemd/settings/units/restrictsuidsgid/","tags":["configuration","linux","sandboxing","service hardening","systemd"],"title":"RestrictSUIDSGID setting"},{"categories":null,"contents":"Why and when to use RestrictNamespaces Systemd provides the setting RestrictNamespaces which aims to restrict what namespaces can be used. Linux namespaces create an abstraction layer around a global resource, such as a filesystem, to isolate it from other processes.\nRelevant namespaces include:\ncgroup ipc mnt net pid user uts When using this option, the default is that it is used as an allow-list and define which namespaces are allowed.\nTo know if a process uses namespaces, consider looking at the source code for the following syscalls:\nclone(2) setns(2) unshare(2) An alternative is to inspect a binary with the strings command or analyze a process with strace.\nSettings This setting may be used once or multiple times. When it is used multiple times, it adds to the list (logical OR). Lines that have a value with tilde (~) prepended, will turn it into AND, blocking the relevant items.\nTo create a deny-list, add only those namespaces to block by prepending the list with a tilde (~).\nWhen this setting is not configured, there are no restrictions when it comes to namespaces that a service can use.\nCaveats This setting only works on:\nx86, x86-64 mips, mips-le, mips64, mips64-le, mips64-n32, mips64-le-n32 ppc64, ppc64-le s390, s390x Generic advice Using this option depends really on the type of service and if a restriction is needed. Normally namespaces may already help shielding a service from a global resource. So this option should be used with care.\n","permalink":"https://linux-audit.com/systemd/settings/units/restrictnamespaces/","tags":["configuration","linux","namespaces","sandboxing","service hardening","systemd"],"title":"RestrictNamespaces setting"},{"categories":null,"contents":"Why and when to use CapabilityBoundingSet Systemd provides the setting CapabilityBoundingSet which aims to limit the capabilities of a process or its children. This powerful filter restricts what a Linux process can do greatly, but requires some in-depth knowledge of the related process and its child processes.\nSee the Linux capabilities 101 for an introduction, as well the overview of capabilities for more details.\nConfiguration This setting takes a space-separated list and may be specified multiple times. If specified multiple times, the capabilities are stacked.\nTo reset an earlier defined set, use \u0026lsquo;CapabilityBoundingSet=~\u0026rsquo;.\nWhen prefixed with ~, inversion of the setting is applied. So defined capabilities will be denied.\nGeneric advice This setting is a powerful option to restrict what processes can or can\u0026rsquo;t do. It requires more than average knowledge from a system administrator, but with some research it is typically possible to get a working and hardened service. Use with caution on active production systems, first test on non-critical systems.\nImplementation and examples When implementing this option, there are multiple methods to discover what capabilities are required:\nLook at binary and see if capability is linked (e.g. using filecap, pscap) Block all capabilities, look at errors in journal Look at syscall usage and map them against a capability To find out what capabilities might be used within an application, consult How to harden a systemd service unit for additional tips.\nExample testing using systemd-run $ touch test $ systemd-run --pty --property=CapabilityBoundingSet=~CAP_CHOWN chown michael:michael test This command should fail, as the CAP_CHOWN capability is not allowed\n","permalink":"https://linux-audit.com/systemd/settings/units/capabilityboundingset/","tags":["capabilities","configuration","linux","sandboxing","syscall","systemd"],"title":"CapabilityBoundingSet setting"},{"categories":["Kernel"],"contents":"This sysctl key defines if the system should create an audit event when a binary is successfully executed, as part of IPE security module.\n","permalink":"https://linux-audit.com/kernel/sysctl/ipe/ipe.success_audit/","tags":["kernel","kernel hardening","linux","sysctl"],"title":"Sysctl: ipe.success_audit"},{"categories":["Kernel"],"contents":"This sysctl key defines if a system should run the Linux security module IPE in permissive or enforce mode.\n","permalink":"https://linux-audit.com/kernel/sysctl/ipe/ipe.enforce/","tags":["kernel","kernel hardening","linux","sysctl"],"title":"Sysctl: ipe.enforce"},{"categories":["Kernel","System Administration"],"contents":"The Linux kernel provides special files in /dev to access physical devices or pseudo-devices. Two character special files within this directory are /dev/random and /dev/urandom . Both are providing random data from the Linux kernel random number generator function, but with some minor differences.\nDifferences between random and urandom The biggest difference between the two random number generator files /dev/random and /dev/urandom is the quality of the random data.\nThe source of /dev/random provides a higher quality, at the cost of less data available. This may result in some delays to get enough data, as the kernel needs to replenish it. The data coming from /dev/urandom is created using a pseudorandom number generator that pulls in data from the entropy pool. It can provide much more data, at the cost of true randomness.\nApplications that really require high quality randomness, especially early during the boot, should use /dev/random. On modern Linux kernels and hardware, typically /dev/urandom is the preferred source and sufficient for most applications. The suggested way is using the syscall getrandom(2) to retrieve the data instead of reading the files directly. This function will actually use /dev/urandom as its default source. The special file /dev/random is considered to be legacy interface and was especially important in a time that the underlying cryptographic implementation of /dev/urandom was less trusted. With changes over time, the trust of the implementation and data in the latter went up.\nMaximum data with read(2) While reading directly from the special files is typically not needed for applications, the underlying read(2) function sees a big difference when reading from both sources. Since kernel 3.16, each read(2) operation will return 512 bytes from /dev/random, while /dev/urandom may return up to 32 MB.\n","permalink":"https://linux-audit.com/kernel/faq/what-is-the-difference-between-dev-random-and-urandom/","tags":["faq","linux","kernel","random"],"title":"What is the difference between /dev/random and /dev/urandom?"},{"categories":null,"contents":"Why and when to use ProtectKernelTunables The systemd unit setting ProtectKernelTunables aims to protect Linux kernel information and settings, so-called kernel tunables. These tunables are stored within /proc and /sys.\nThis setting will also restrict access fully to /proc/kallsyms as well as /proc/kcore .\n/proc/kallsymscontains a symbol table from the Linux kernel providing function and variable names. It may be used by developers and system administrators to troubleshoot kernel issues.\nSee What is the file /proc/kallsyms on Linux? for more details.\nValid options The value no is the default setting and will not restrict access. Use yes to mark most of the Linux kernel tunables read-only and restrict access to the /proc/kallsyms as well as /proc/kcore .\nGeneric advice For most services the value ProtectKernelTunables=yes can be used to restrict access to the kernel tunables.\n","permalink":"https://linux-audit.com/systemd/settings/units/protectkerneltunables/","tags":["configuration","linux","procfs","sandboxing","service hardening","systemd"],"title":"ProtectKernelTunables setting"},{"categories":null,"contents":"What information does /proc/kallsyms contain? The file has three columns of information:\nMemory address of the symbol Symbol type Symbol name Purpose By using the information from the /proc/kallsyms file, a developer or system administrator may troubleshoot issues, such as kernel panics. When a kernel panic happened, a so-called core dump is generated. This file contains in-depth details, such as memory addresses where the error happened. The information from the core dump can be compared with the addresses in /proc/kallsyms to identify the function or variable which caused the issue.\nAnother purpose of the information in /proc/kallsyms is for finding memory leaks. The stored symbols can be inspected and may give developers a clue if a particular piece of code is not correctly handing memory, which may result in a memory leak.\nSecurity considerations to take in account Providing access to /proc/kallsyms may result in exposing sensitive information about the kernel. This is due to its content and in particular the internals of the kernel, such as memory information. Typically only superusers should have access. For most services and users it is better to deny access to this file.\n","permalink":"https://linux-audit.com/what-is/proc-kallsyms/","tags":["core dump","linux","proc","procfs","what-is"],"title":"What is the file /proc/kallsyms on Linux?"},{"categories":null,"contents":"Why and when to use LockPersonality The systemd unit setting LockPersonality prevents changing the personality with personality(2). This is a syscall that defines the kernel execution domain for a process. Normally this kernel execution domain is set to default, unless specified with the Personality= setting.\nConfiguration options of LockPersonality When this unit setting is set to \u0026lsquo;yes\u0026rsquo;, no changes in the personality are allowed.\nGeneric advice Most services can be configured with LockPersonality=yes.\n","permalink":"https://linux-audit.com/systemd/settings/units/lockpersonality/","tags":["configuration","linux","service hardening","systemd"],"title":"LockPersonality setting"},{"categories":null,"contents":"Why and when to use NoNewPrivileges The systemd unit setting NoNewPrivileges prevents processes and its children of obtaining new privileges. Normally this is possible via execve(2), a syscall that executes a program and when filesystem capabilities provide new privileges. Another option is obtaining this via setgid and setuid bits on files.\nConfiguration options of NoNewPrivileges When this unit setting is set to \u0026lsquo;yes\u0026rsquo;, the process and child processes will be denied the possibility to get additional privileges.\nGeneric advice Most services can be configured with NoNewPrivileges=yes.\n","permalink":"https://linux-audit.com/systemd/settings/units/nonewprivileges/","tags":["binary","configuration","linux","sandboxing","service hardening","systemd"],"title":"NoNewPrivileges setting"},{"categories":null,"contents":"Why and when to use SystemCallArchitectures Usually Linux user processes talk via an interface with an operating system facility using a so-called ABI . For common instructions this is done using a syscall. When a system supports multiple ABIs, it may be useful to restrict the set that can be used to prevent circumventing a systemd setting like SystemCallFilter. The setting SystemCallArchitectures can be used to restrict this.\nGeneric advice For most systemd units, the setting SystemCallArchitectures=native is advised to restrict access one set of CPU instructions.\n","permalink":"https://linux-audit.com/systemd/settings/units/systemcallarchitectures/","tags":["configuration","linux","sandboxing","service hardening","syscall","systemd"],"title":"SystemCallArchitectures setting"},{"categories":null,"contents":"Why and when to use PrivateDevices By default, a process can see most of the devices in /dev and interact with these devices. The PrivateDevices unit setting in systemd restricts the list of devices. Only pseudo-devices such as /dev/null are made available to the process.\nGeneric advice For most systemd units the setting PrivateDevices=yes can be safely used.\nTesting To see how this setting impacts a process, consider using the systemd-run command.\nsystemd-run --pty --property=PrivateDevices=yes ls -l /dev total 0 drwxr-xr-x 2 root root 180 nov 15 10:03 char lrwxrwxrwx 1 root root 11 nov 15 10:03 core -\u0026gt; /proc/kcore lrwxrwxrwx 1 root root 13 nov 15 10:03 fd -\u0026gt; /proc/self/fd crw-rw-rw- 1 root root 1, 7 nov 15 10:03 full drwxr-xr-x 3 root root 0 sep 3 09:52 hugepages lrwxrwxrwx 1 root root 28 nov 15 10:03 log -\u0026gt; /run/systemd/journal/dev-log drwxrwxrwt 2 root root 40 sep 3 09:51 mqueue crw-rw-rw- 1 root root 1, 3 nov 15 10:03 null crw-rw-rw- 1 root root 5, 2 nov 15 10:03 ptmx drwxr-xr-x 2 root root 0 sep 3 09:51 pts crw-rw-rw- 1 root root 1, 8 nov 15 10:03 random drwxrwxrwt 4 root root 280 nov 15 10:03 shm lrwxrwxrwx 1 root root 15 nov 15 10:03 stderr -\u0026gt; /proc/self/fd/2 lrwxrwxrwx 1 root root 15 nov 15 10:03 stdin -\u0026gt; /proc/self/fd/0 lrwxrwxrwx 1 root root 15 nov 15 10:03 stdout -\u0026gt; /proc/self/fd/1 crw-rw-rw- 1 root root 5, 0 nov 15 10:03 tty crw-rw-rw- 1 root root 1, 9 nov 15 10:03 urandom crw-rw-rw- 1 root root 1, 5 nov 15 10:03 zero In the output only the new pseudo-devices will be shown now.\n","permalink":"https://linux-audit.com/systemd/settings/units/privatedevices/","tags":["configuration","devices","linux","service hardening","systemd"],"title":"PrivateDevices setting"},{"categories":null,"contents":"Why and when to use PrivateTmp By default, a process can see the content of /tmp and /var/tmp, if the owner or group has the right file permissions. By using the setting PrivateTmp, access may be reduced to these temporary directories. Systemd does this by defining a new namespace for the process.\nGeneric advice For most services PrivateTmp=yes can be used. It is an easy way to reduce risks related to temporary files.\nWhen multiple processes need to access the same files, then typically this should be avoided in the temporary directories, but create a specific application directory those processes to exchange data.\nTesting To see if a program works with this property, consider using the systemd-run command.\nMake sure that there is a file in /tmp, then run the command with PrivateTmp set.\nsystemd-run --pty --property=PrivateTmp=yes ls -l /tmp total 0 No files are available now, as the ls program only sees content from the new namespace.\n","permalink":"https://linux-audit.com/systemd/settings/units/privatetmp/","tags":["configuration","linux","sandboxing","service hardening","systemd"],"title":"PrivateTmp setting"},{"categories":null,"contents":"Why and when to use ExecPaths The systemd setting ExecPaths defines the paths that are allowed for program execution. It is the opposite of NoExecPaths and they are typically used together. A common option is first to disable execution from \u0026lsquo;/\u0026rsquo; with NoExecPaths, then define the paths in ExecPaths where the related binaries of the service are located.\nConfiguration options of ExecPaths Define the paths that are allowed for program execution. This is typically done after a more generic \u0026lsquo;block\u0026rsquo; has been defined using NoExecPaths, such as the root path.\nExample Execution is only allowed if the binary is /usr/sbin/myprogram and libraries from /usr/lib.\nNoExecPaths=/ ExecPaths=/usr/lib /usr/sbin/myprogram Testing To see if a program works with this property, consider using the systemd-run command.\nsystemd-run --pty --property=NoExecPaths=/tmp --property=ExecPaths=/tmp/testing /tmp/testing/mytestfile\nFor this example to work correctly, make sure that the path /tmp/testing does exist. The mytestfile could be something like a copy of the /bin/ps file.\n","permalink":"https://linux-audit.com/systemd/settings/units/execpaths/","tags":["binary","configuration","linux","processes","sandboxing","service hardening","systemd"],"title":"ExecPaths setting"},{"categories":null,"contents":"Why and when to use NoExecPaths The setting NoExecPaths reduces program execution from paths that are specified. By combining it with ExecPaths, the execution can be greatly restricted.\nConfiguration options of NoExecPaths Define one or more paths that should be restricted. Use \u0026lsquo;/\u0026rsquo; to completely disallow all paths.\nAdd a \u0026lsquo;-\u0026rsquo; (minus) to the beginning of the path to ignore the path if it does not exist. Use a \u0026lsquo;+\u0026rsquo; (plus) to make it relative to the root directory of the unit.\nGeneric advice For most services NoExecPaths can be used if correctly combined with ExecPaths.\nNoExecPaths example Execution is only allowed if the binary is /usr/sbin/myprogram and libraries from /usr/lib.\nNoExecPaths=/ ExecPaths=/usr/lib /usr/sbin/myprogram Testing To see if a program works with this property, consider using the systemd-run command.\nsystemd-run --pty --property=NoExecPaths=/tmp /tmp/testfile\n","permalink":"https://linux-audit.com/systemd/settings/units/noexecpaths/","tags":["configuration","linux","processes","service hardening","systemd"],"title":"NoExecPaths setting"},{"categories":null,"contents":"Why and when to use ProtectControlGroups The systemd unit setting ProtectControlGroups reduces write access to cgroup or Linux control groups. Information about cgroups are normally available under /sys/fs/cgroup. This setting may restrict a process from writing anything to this directory structure.\nConfiguration options Before systemd 257, only boolean values (yes/no, true/false) were accepted. With systemd 257 private and strict where added.\nGeneric advice For most services ProtectControlGroups can be turned on. Only container managers do require write access to the control groups structures.\nExample configuration [Service] ProtectControlGroups=yes ","permalink":"https://linux-audit.com/systemd/settings/units/protectcontrolgroups/","tags":["cgroups","configuration","linux","sandboxing","service hardening","systemd"],"title":"ProtectControlGroups setting"},{"categories":null,"contents":"Why and when to use ProtectSystem The setting ProtectSystem reduces write access to specific paths based on the setting.\nGeneric advice For most services ProtectSystem can be used with yes, full, or strict. The latter is the most strict setting and often requires ReadWritePaths to allow application-specific paths to work.\nTesting Use the systemd-run command to see if file creation is restricted by setting it to \u0026lsquo;strict\u0026rsquo;.\nsystemd-run --pty --property=ProtectSystem=strict touch ./testfile\nThis should return an error:\n/usr/bin/touch: cannot touch \u0026#39;./testfile\u0026#39;: Read-only file system Example configuration Values Define the paths that are granted write access.\n[Service] ProtectSystem=strict ReadWritePaths=/var/log/application-logs /var/run/application Caveats of ProtectSystem The option ProtectSystem has no effect when:\nkernel has no file system namespacing service manager within a container manager that makes file system namespacing unavailable ","permalink":"https://linux-audit.com/systemd/settings/units/protectsystem/","tags":["configuration","linux","sandboxing","service hardening","systemd"],"title":"ProtectSystem setting"},{"categories":null,"contents":"Why and when to use ProtectClock The setting ProtectClock reduces access the clock information.\nConfiguration options of ProtectClock When this setting is not configured or disabled (e.g. no) access to clock information is not restricted. When set (e.g. true), the service will no longer have access to clock information. A system function call like clock_adjtime(2) will fail.\nGeneric advice For most services ProtectClock=yes can be used, unless it really depends on retrieving clock information.\nTesting To see if a program works with this property, consider using the systemd-run command.\nsystemd-run --pty --property=ProtectClock=yes /path/to/PROGRAM\nAnother option is to use strace and monitor for syscalls related to the clock.\n#include \u0026lt;stdio.h\u0026gt; #include \u0026lt;sys/timex.h\u0026gt; int main(int argc, char *argv[]) { struct ntptimeval timestate = {0}; // Try to get the time using the syscall clock_adjtime ntp_gettime(\u0026amp;timestate); // Returns Success if it worked perror(\u0026#34;clock_adjtime\u0026#34;); } Compile the program using gcc:\ngcc protectclock.c\nThen run it:\n./a.out\nTo look at the syscalls being used:\nstrace ./a.out\nExample when the clock is accessible:\nclock_adjtime(CLOCK_REALTIME, {modes=0, offset=479672, freq=634399, maxerror=175500, esterror=0, status=STA_PLL|STA_NANO, constant=7, precision=1, tolerance=32768000, time={tv_sec=1731537097, tv_usec=153901179}, tick=10000, ppsfreq=0, jitter=0, shift=0, stabil=0, jitcnt=0, calcnt=0, errcnt=0, stbcnt=0, tai=0}) = 0 (TIME_OK) ","permalink":"https://linux-audit.com/systemd/settings/units/protectclock/","tags":["clock","configuration","linux","ntp","sandboxing","service hardening","systemd","time"],"title":"ProtectClock setting"},{"categories":["System Administration"],"contents":"The journalctl command can be used to remove older journal logs by using the --vacuum-time option, followed by a time. This time defines the maximum age of entries and items that are too old will be purged from the logs. This is a useful option to regain disk space when the logs are using too much of it.\nUsage Before clearing out any journal logs, it may be useful to see the current disk usage first. Use the --disk-usage option to show\n# journalctl --disk-usage Archived and active journals take up 256.0M in the file system. To remove all journal logs older than 30 days, define \u0026lsquo;30d\u0026rsquo; as its period.\n# journalctl --vacuum-time=30d Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-0000000000022eb1-000618f16cbeab92.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-0000000000026d71-00061acf10c2969d.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000271d2-00061aeda8cf5fdd.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-00000000000271ea-00061aedb700b831.journal (24.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000271eb-00061aedb7013c8a.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-000000000002bfcd-00061d5203640666.journal (16.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-000000000002d64e-00061e085359feff.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-000000000002f181-00061eaaa9badf82.journal (16.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-000000000003237e-0006203b3f8d0893.journal (16.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000341a1-00062132520d58aa.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-00000000000354b9-000621d042fce1c1.journal (16.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-0000000000035a59-000621ff787d8caf.journal (8.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/system@ca889eb2eae24e41b37a50d33bad131c-0000000000038580-0006236a1be25d21.journal (16.0M). Deleted archived journal /var/log/journal/d8da4544b42732906a930deaba49a506/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000395fc-000623f52ec71ab7.journal (8.0M). Vacuuming done, freed 168.0M of archived journals from /var/log/journal/d8da4544b42732906a930deaba49a506. Vacuuming done, freed 0B of archived journals from /var/log/journal. Vacuuming done, freed 0B of archived journals from /run/log/journal. ","permalink":"https://linux-audit.com/systemd/faq/how-to-clear-systemd-journal-logs-by-time-period/","tags":["faq","journalctl","linux","systemd"],"title":"How to clear systemd journal logs by time"},{"categories":["File systems","System Administration"],"contents":"To find out what processes are actively using the disk, we need to monitor disk I/O. Commands that can be used include iostat, iotop, and vmstat.\nExamples using iotop Run the iotop command without any parameters to see an interactive display of the total disk read and write speeds, including the processes responsible for initiating these read or write actions.\n# iotop Total DISK READ: 0.00 B/s ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ | Total DISK WRITE: 0.00 B/s Current DISK READ: 0.00 B/s ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ | Current DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE GRAPH[R+W]▽ COMMAND [T] 1 be/4 root 0.00 B/s 0.00 B/s systemd ▲ 2 be/4 root 0.00 B/s 0.00 B/s kthreadd █ 3 be/0 root 0.00 B/s 0.00 B/s rcu_gp █ 4 be/0 root 0.00 B/s 0.00 B/s rcu_par_gp █ 5 be/0 root 0.00 B/s 0.00 B/s slub_flushwq █ 6 be/0 root 0.00 B/s 0.00 B/s netns █ Use \u0026lsquo;q\u0026rsquo; to quit.\nNon-interactive usage of iotop Another option is to run iotop non-interactively. This is achieved by using the --batch option. To make it automatically stop after a few times, we can provide a the --iter=NUMBER option.\n# iotop --batch --iter=3 To reduce the output further, we can tell the program to queue the activity by adding a delay with --delay=SECONDS. Instead of a default delay of a single second, we can take a longer period. We need at least two iterations to gather the information. Typically only the active processes are interesting, which we can filter out using the --only option.\n# iotop --batch --only --delay=30 --iter=2 Disk activity per device vmstat To learn about the most active devices, use the vmstat command. To see a single moment in time, run the command without any parameters.\nvmstat\nTo queue up disk activity and get a better idea on the most active device, let the tool perform two measures with a time interval (e.g. 30 seconds).\n# vmstat --disk 30 2 disk- ------------reads------------ ------------writes----------- -----IO------ total merged sectors ms total merged sectors ms cur sec loop0 276 0 3446 32 0 0 0 0 0 0 loop1 1525 0 131252 192 0 0 0 0 0 4 loop2 361 0 10116 97 0 0 0 0 0 0 loop3 247 0 3484 29 0 0 0 0 0 0 loop4 410 0 10748 141 0 0 0 0 0 0 loop5 404 0 7118 48 0 0 0 0 0 0 loop6 175 0 2276 23 0 0 0 0 0 0 loop7 0 0 0 0 0 0 0 0 0 0 sr0 5 0 2 0 0 0 0 0 0 0 sda 311804 43728 22006273 97600 1922715 958862 116556904 1651642 0 1913 dm-0 341315 0 19843630 120412 2847459 0 113510856 2880140 0 1894 loop0 276 0 3446 32 0 0 0 0 0 0 loop1 1525 0 131252 192 0 0 0 0 0 4 loop2 361 0 10116 97 0 0 0 0 0 0 loop3 247 0 3484 29 0 0 0 0 0 0 loop4 410 0 10748 141 0 0 0 0 0 0 loop5 404 0 7118 48 0 0 0 0 0 0 loop6 175 0 2276 23 0 0 0 0 0 0 loop7 0 0 0 0 0 0 0 0 0 0 sr0 5 0 2 0 0 0 0 0 0 0 sda 311804 43728 22006273 97600 1922737 958875 116557312 1651690 0 1913 dm-0 341315 0 19843630 120412 2847494 0 113511264 2880252 0 1894 iostat Another tool that provides insights like vmstat is the the iotop command. By default with colored output and easier to read. Like with other tools we can define a delay and number of measurements.\n# iostat 30 2 avg-cpu: %user %nice %system %iowait %steal %idle 0.20 0.01 0.04 0.01 0.01 99.73 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd dm-0 0.34 1.04 5.92 8.96 9922115 56756396 85913152 loop0 0.00 0.00 0.00 0.00 1723 0 0 loop1 0.00 0.01 0.00 0.00 65626 0 0 loop2 0.00 0.00 0.00 0.00 5058 0 0 loop3 0.00 0.00 0.00 0.00 1742 0 0 loop4 0.00 0.00 0.00 0.00 5374 0 0 loop5 0.00 0.00 0.00 0.00 3559 0 0 loop6 0.00 0.00 0.00 0.00 1138 0 0 sda 0.24 1.15 6.08 9.42 11003436 58279420 90309104 sr0 0.00 0.00 0.00 0.00 1 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.43 0.00 0.17 0.10 0.03 99.27 Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd dm-0 1.23 0.00 8.00 0.00 0 240 0 loop0 0.00 0.00 0.00 0.00 0 0 0 loop1 0.00 0.00 0.00 0.00 0 0 0 loop2 0.00 0.00 0.00 0.00 0 0 0 loop3 0.00 0.00 0.00 0.00 0 0 0 loop4 0.00 0.00 0.00 0.00 0 0 0 loop5 0.00 0.00 0.00 0.00 0 0 0 loop6 0.00 0.00 0.00 0.00 0 0 0 sda 0.73 0.00 8.00 0.00 0 240 0 sr0 0.00 0.00 0.00 0.00 0 0 0 When reviewing the data, be aware that the first set of data shows a total number per device. The second one shows the difference, which makes it a great tool to see what happened in the last 30 seconds.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-monitor-disk-io/","tags":["faq","file system","forensics","howto","linux","processes"],"title":"How to monitor disk activity (I/O) on Linux"},{"categories":["File systems","System Administration"],"contents":"The file command is available on Linux systems to retrieve file information. It can determine the file type by analyzing specific sections of the file and return the most likely file type. The first few lines of a file often reveal the file type. But since a file can be binary data, it may mess up the terminal output when using the cat command.\nExamples using file The /etc/passwd file is a good target to see an example of a plain text file.\n# file /etc/passwd /etc/passwd: ASCII text To see the output of a binary file, we can inspect /bin/ls .\n# file /bin/ls /bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=36b86f957a1be53733633d184c3a3354f3fc7b12, for GNU/Linux 3.2.0, stripped Depending on the file, additional details are provided. As can be seen in the example above, it does not just show it is an executable file (ELF), but also the processor platform, how it is linked, a build hash, and that debug symbols are stripped.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-file-type/","tags":["elf","faq","file system","howto","linux"],"title":"How to see the file type on Linux"},{"categories":["File systems","System Administration"],"contents":"The ls command is the default command to see basic file properties, such as its name, ownership, file permissions, and file size. When using the -l the long listing format is used to see the details.\nShowing file properties # ls -l /etc/passwd -rw-r--r-- 1 root root 1993 Feb 2 2024 /etc/passwd When a file is more than a megabyte, it may help to use the --human-readable option. It show the size with a corresponding character (k=kilobyte, m=megabyte, g=gigabyte).\nAn alternative option is using the stat command. It will provide additional details in its default output, such as the initial creation date and inode.\n# stat /etc/passwd File: /etc/passwd Size: 1993 Blocks: 8 IO Block: 4096 regular file Device: fd00h/64768d\tInode: 132674 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-10 19:17:01.327124248 +0000 Modify: 2024-02-02 19:14:45.316650111 +0000 Change: 2024-02-02 19:14:45.320650046 +0000 Birth: 2024-02-02 19:14:45.316650111 +0000 ","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-file-size/","tags":["faq","file system","howto","linux"],"title":"How to see the size of a file"},{"categories":["File systems","System Administration"],"contents":"Use the ls command with -t option to sort files by its modification time. The newest files or directories will be shown at the top. To show the a sorted list with the oldest first, use ls -lrt. This reverses the output, starting with the oldest files and ending with the most recently changed ones.\nChanging ls output when sorting Linux uses the GNU version of ls with additional options to sort and display directory contents.i\nReverse sorting To see the oldest files first, use the -r option.\nls -lrt\nFormat of modification time (mtime) The ls command has different ways of showing the modification time, depending on the age of the file. When sorting files it may be easier to see normalize this date and time, by using the ISO 8601 format.\n# ls -lt --time-style=long-iso total 117 -rwxrwxr-x 1 michael michael 32 2024-07-08 09:47 file1.yaml -rwxrwxr-x 1 michael michael 43 2024-06-27 17:42 file2.yaml -rwxrwxr-x 1 michael michael 41 2024-06-27 17:40 file3.yaml -rwxrwxr-x 1 michael michael 46 2024-06-27 17:40 file4.yaml -rwxrwxr-x 1 michael michael 27 2024-06-21 13:00 file5.yaml -rwxrwxr-x 1 michael michael 62 2024-06-14 18:07 file6.yaml \u0026lt;snip\u0026gt; To see even more detailed output in the time field, use \u0026lsquo;full-iso\u0026rsquo; as the time style.\n# ls -lt --time-style=full-iso total 117 -rwxrwxr-x 1 michael michael 32 2024-07-08 09:47:18.374452164 +0000 file1.yaml -rwxrwxr-x 1 michael michael 43 2024-06-27 17:42:09.686911617 +0000 file2.yaml -rwxrwxr-x 1 michael michael 41 2024-06-27 17:40:28.984909993 +0000 file3.yaml -rwxrwxr-x 1 michael michael 46 2024-06-27 17:40:15.844909781 +0000 file4.yaml -rwxrwxr-x 1 michael michael 27 2024-06-21 13:00:00.348638620 +0000 file5.yaml -rwxrwxr-x 1 michael michael 62 2024-06-14 18:07:11.121935846 +0000 file6.yaml \u0026lt;snip\u0026gt; This output itself can also be reversed as before.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-display-directory-contents-sorted-by-modification-time/","tags":["faq","file system","howto","linux","ls"],"title":"How to display directory contents sorted by modification time"},{"categories":["System Administration"],"contents":"Systemd uses different types of units. One of them is the timer unit, which can be used to schedule a periodic task. This timer unit is linked to an existing service unit and will activate the service according to the defined schedule. The timer unit can be defined using the information about systemd timers.\nTimers use one or more OnCalendar definitions to specify when execution of the timer should happen. As systemd timers are very versatile and provide good monitoring options, they are a good replacement of cronjobs. The only downside is that slightly more configuration is needed when a just a single shell script needs to be executed, as you need to create both the timer as service unit.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-schedule-a-periodic-task-with-systemd/","tags":["faq","linux","systemd"],"title":"How to schedule a periodic task with systemd"},{"categories":["System administration"],"contents":"","permalink":"https://linux-audit.com/system-administration/commands/systemd-analyze/","tags":["dns","elf","linux","systemd"],"title":"systemd-analyze"},{"categories":["System Administration"],"contents":"Systemd is nowadays a common system and service manager for Linux systems. But how do you know for sure that it is being active? The easiest way is to have a look at PID number 1. This is the first process started after the kernel itself. With the help of ps we can determine the underlying command behind this initial process.\nps -p 1 -o comm=\nThis command defines what columns should be part of the output, where only shows the actual command. Adding the \u0026lsquo;=\u0026rsquo; removes the header, so if systemd is being used, then the string \u0026lsquo;systemd\u0026rsquo; will be returned.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-check-if-systemd-is-used-or-active/","tags":["faq","linux","systemd"],"title":"How to check if systemd is being used or active"},{"categories":["System Administration"],"contents":"When the SSH agent is running, the ssh-add command can be used to load a SSH key. The SSH agent then will request a password when needed, and load the key details in memory.\nAdding the SSH key just requires the path to the private key.\nssh-add ~/.ssh/id_ed25519\nWhen the key is loaded, use the -l or -L option to show the identities that the SSH agent has.\nssh-add -l\nErrors Could not open a connection to your authentication agent Most likely your SSH agent is not running. Check if the process is running. If not, start the agent:\neval $(ssh-agent)\n","permalink":"https://linux-audit.com/ssh/faq/how-to-add-ssh-key-to-ssh-agent/","tags":["authentication","ed25519","faq","howto","linux","openssh","ssh","ssh-agent"],"title":"How to add a SSH key to the SSH agent"},{"categories":["System Administration"],"contents":"The systemctl command can be used to show active units. To see only the services that are enabled, we need to filter the output. This can be achieved using the list-unit-files subcommand and combined with the --state= option. As we are interested in enabled services only, set the value to enabled.\nWant to parse the output with a script? Consider adding --legend=false to remove the header and footer text (X unit files listed.).\nUsage systemctl list-unit-files --type=service --state=enabled UNIT FILE STATE VENDOR PRESET apparmor.service enabled enabled blk-availability.service enabled enabled cloud-config.service enabled enabled cloud-final.service enabled enabled cloud-init-local.service enabled enabled cloud-init.service enabled enabled console-setup.service enabled enabled cron.service enabled enabled dmesg.service enabled enabled e2scrub_reap.service enabled enabled finalrd.service enabled enabled getty@.service enabled enabled gpu-manager.service enabled enabled grub-common.service enabled enabled grub-initrd-fallback.service enabled enabled irqbalance.service enabled enabled keyboard-setup.service enabled enabled lvm2-monitor.service enabled enabled lxd-agent.service enabled enabled ModemManager.service enabled enabled multipathd.service enabled enabled networkd-dispatcher.service enabled enabled nginx.service enabled enabled open-iscsi.service enabled enabled open-vm-tools.service enabled enabled pollinate.service enabled enabled rsyslog.service enabled enabled secureboot-db.service enabled enabled setvtrgb.service enabled enabled snap.lxd.activate.service enabled enabled snapd.apparmor.service enabled enabled snapd.autoimport.service enabled enabled snapd.core-fixup.service enabled enabled snapd.recovery-chooser-trigger.service enabled enabled snapd.seeded.service enabled enabled snapd.service enabled enabled snapd.system-shutdown.service enabled enabled ssh.service enabled enabled systemd-networkd-wait-online.service enabled disabled systemd-networkd.service enabled enabled systemd-pstore.service enabled enabled systemd-resolved.service enabled enabled systemd-timesyncd.service enabled enabled thermald.service enabled enabled ua-reboot-cmds.service enabled enabled ubuntu-advantage.service enabled enabled udisks2.service enabled enabled ufw.service enabled enabled unattended-upgrades.service enabled enabled vgauth.service enabled enabled 50 unit files listed. ","permalink":"https://linux-audit.com/systemd/faq/how-to-see-all-enabled-services-with-systemctl/","tags":["faq","linux","systemctl","systemd"],"title":"How to see all enabled services with systemctl"},{"categories":null,"contents":"Introduction This is a hardening profile to help securing nginx by using systemd unit configuration. It\u0026rsquo;s goal is to restrict what nginx can do and make it harder for any possible vulnerability to be misused.\nThe rationale for the selected settings is based on the analysis as part of the article Hardening nginx with systemd security features.\nRelevant FAQ: How to use systemctl edit to change a service?\nNotes and fine-tuning As nginx can be tailored to multiple needs, most likely some fine-tuning is needed. Depending on external components such as PHP, temporary files may need to be made accessible. When possible, define an alternative directory and keep PrivateTmp enabled.\nIf you are using an older hardening profile, please upgrade at least to 0.5 or higher.\nDebian Debian systems may need an extension to ExecPaths and allow also /sbin/start-stop-daemon\nTransparent proxy When using proxy functionality (proxy_bind $remote_addr transparent), adjustments may need to be made to allow this. Otherwise nginx can\u0026rsquo;t fork the worker processes.\nSecureBits=no-setuid-fixup-locked noroot-locked CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_CHOWN CAP_SETGID CAP_SETUID CAP_NET_RAW Rationale: nginx uses prctl(PR_SET_KEEPCAPS, 1) before setuid(). If keep-caps-locked is set in SecureBits, an EPERM issue will occur with a fatal exit 2 code for the worker processes.\nGot a more tight version that works? Let it know.\nChanges Adjust SecureBits (0.5) Add PrivateTmp=yes Changed LockPersonality from \u0026rsquo;true\u0026rsquo; to \u0026lsquo;yes\u0026rsquo; Additional comments Add capability: CAP_SETPCAP due to usage of SecureBits Add SecureBits Allow real-time scheduling Special thanks To Jean-Baptiste for reporting an issue with SecureBits that may allow access to files while normally not permitted with file permissions.\n","permalink":"https://linux-audit.com/systemd/hardening-profiles/nginx/","tags":["configuration","hardening","linux","nginx","systemd"],"title":"Nginx hardening profile"},{"categories":null,"contents":"Why and when to use SocketBindAllow The setting SocketBindAllow is used together with SocketBindDeny and defines restrictions on the usage of the system call bind(2) on a network socket.\nSettings Both SocketBindAllow and SocketBindDeny use a bind-rule. See SocketBindDeny for the details.\nGeneric advice This setting is useful in combination with SocketBindDeny to create an allow-list.\nExamples Allow binding on TCP port 80 can be defined by first deny all protocols and ports, followed by the allowed protocol and port combination.\n[Service] SocketBindDeny=any SocketBindAllow=tcp:80 Allow binding on port 443 on all protocols (IPv4/IPv6, TCP/UDP).\n[Service] SocketBindDeny=any SocketBindAllow=443 ","permalink":"https://linux-audit.com/systemd/settings/units/socketbindallow/","tags":["configuration","linux","networking","sandboxing","service hardening","systemd"],"title":"SocketBindAllow setting"},{"categories":null,"contents":"Why and when to use SocketBindDeny The setting SocketBindDeny can be used alone or together with SocketBindAllow to set restrictions on the usage of the system call bind(2) on a network socket.\nSettings If the SocketBindDeny list is used alone, then it is a deny-list. Everything except the defined ports/protocols will be allowed.\nBy defining the value \u0026lsquo;any\u0026rsquo;, all combinations are denied. This is typically used in combination with SocketBindAllow to open up one or more ports.\nBoth SocketBindAllow and SocketBindDeny use a bind-rule in the following format: [address-family:][transport-protocol:][ip-ports] | any\nAddress family Optional, will match all address families if not specified. If specified, then the valid options for the address family are:\nipv4 ipv6 Transport protocol Optional, while match all transport protocol if not specified. If specified, then the valid options for the transport protocol are:\ntcp udp IP ports IP ports can be specified as a single port or a range, optionally with an address family or transport protocol. The IP port or port range is also optional, as SocketBindAllow and SocketBindDeny allow specifying one of the other filter alone.\nAny The \u0026lsquo;any\u0026rsquo; keyword is reserved as a wildcard. It will match all combinations of address families, transport protocols, and ports.\nGeneric advice This setting is powerful to restrict what ports or protocols can be used. When possible set the value to \u0026lsquo;any\u0026rsquo; and then open ports with SocketBindAllow. This way ports are restricted and are a good foundation for firewall rules.\nExamples Deny all, except for ports 2000 and 3000.\n[Service] SocketBindDeny=any SocketBindAllow=2000 SocketBindAllow=3000 Deny all Used alone (deny-list), this can prevent any bindings.\n[Service] SocketBindDeny=any Block only IPv6 Used alone (deny-list), block all IPv6 ports for binding.\n[Service] SocketBindDeny=ipv6 ","permalink":"https://linux-audit.com/systemd/settings/units/socketbinddeny/","tags":["configuration","ipv6","linux","networking","sandboxing","service hardening","sockets","systemd"],"title":"SocketBindDeny setting"},{"categories":null,"contents":"Why and when to use DeviceAllow Systemd does not restrict access devices by systemd units by default. The setting DeviceAllow aims to reduce device access, typically of those available within /dev. If this setting is defined, then access to devices will be restricted.\nSettings Define DeviceAllow with a path or string and optionally an access level. When using a path, using the full path to a file (/dev/sda1). A string can be used to define a particular type of device (char-rtc) or a group (char-*) by using a wildcard. If DevicePolicy is not specifically configured, access to a few generic devices will be also allowed ( /dev/full , /dev/null , /dev/random , /dev/urandom , /dev/zero ).\nDeviceAllow=/dev/sda3 r Other examples:\nDeviceAllow=/dev/sda1 DeviceAllow=block-* DeviceAllow=char-* DeviceAllow=char-rtc r See /proc/devices for available character and block devices. Prefix them with their class (char or block) like in the examples.\nGeneral advice For most services it might be easier to use PrivateDevices (with value yes) or set DevicePolicy to reduce the devices that can be accessed.\nImplementation To find out what devices might be used by a service, there are multiple ways to discover them:\nHave a look at the source code of the program and look specific entries pointing to /dev Use the Linux audit framework to set a watch on /dev and monitor usage by a process Use the strings command on a binary and filter out all entries starting with /dev ","permalink":"https://linux-audit.com/systemd/settings/units/deviceallow/","tags":["configuration","devices","linux","sandboxing","service hardening","systemd"],"title":"DeviceAllow setting"},{"categories":null,"contents":"Why and when to use DevicePolicy Systemd has the setting DevicePolicy that aims to reduce access to devices in /dev. By default, there is no limitation to access devices.\nSettings The value strict is the most strict, as the name implies. This is suitable for services that do not need any access, like custom shell scripts. With DeviceAllow access to some paths can be granted. See this setting to find out how to determine what devices are being accessed.\nGeneric advice Aim for using \u0026lsquo;strict\u0026rsquo; when possible and define entries that should be allowed. To discover files used by a binary, consider inspecting it with the strings command or look at open files from a running process with lsof.\n","permalink":"https://linux-audit.com/systemd/settings/units/devicepolicy/","tags":["configuration","devices","linux","lsof","sandboxing","service hardening","systemd"],"title":"DevicePolicy setting"},{"categories":["System Administration"],"contents":"Units in systemd may fail for a variety of reasons. In this article some examples are collected to help with troubleshooting them.\nGot an issue that you can\u0026rsquo;t get resolved by using this article? Share it via the contact page and we try to help!\nCommon causes Incorrect file permissions Most services need to read or write data. When an underlying directory or specified file can\u0026rsquo;t be opened (for reading or writing), then the service may exit. Typically this is done with an exit code greater than 0, to indicate that execution of the program was terminated early.\nConfiguration too strict When using the systemd security features to secure units, the configuration might prevent a service from starting as well. Commonly in that case a syscall is blocked or the file permissions or ownership are incorrect.\nGeneric troubleshooting steps Systemd provides several ways of discovering why a service does not want to stop or terminated early or unexpectedly. Typically it will require only a few of the methods below to see when and why this was the case.\nCheck the status of a service systemctl status nginx\nThis output will typically include a few lines of the journal\nConsult the journal for more entries If the output from the status subcommand does not reveal the details, then consider listing all entries of the day.\njournalctl -u nginx --since=\u0026quot;today\u0026quot;\nStill not giving the answer, then query the journal without the unit and show the last 50 lines.\njournalctl -n 50\nAdditional steps to resolve failed units Restart the service The obvious step after making changes it to restart it.\nsystemctl restart nginx.service\nReset the unit Restarting not making a difference? Try resetting the failed unit, as this resets also restart counters.\nsystemctl reset-failed nginx.service\nRestore an altered unit If you made changes to a unit, consider clearing them.\nNot sure if a unit file was changed? Look at the output of the cat subcommand and see if it shows any overrides.\nsystemctl cat nginx.service\nConfiguration changed? Make a copy of the override.conf or any other drop-in files. Then use revert to restore a unit to its original configuration.\nsystemctl revert nginx.service\nSuspecting a blocked syscall? Typically syscalls are blocked due to the use of SystemCallFilter. In that case, seccomp might be responsible for blocking a particular syscall.\nseccomp blocking If seccomp is blocking requests the requests, then they will be logged to the journal. Query the journal with journalctl and select the seccomp entries.\njournalctl _AUDIT_TYPE_NAME=SECCOMP\nExample output:\nDec 10 01:41:46 test audit[15307]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined pid=15307 comm=\u0026#34;smtpd\u0026#34; exe=\u0026#34;/usr/sbin/smtpd\u0026#34; sig=31 arch=c000003e syscall=161 compat=0 ip=0x7fc8600cfb57 code=0x80000000 Use strace Another good method to discover a blocked syscall, is to use the strace command. In that case, disable first the entries that you suspect are causing the issue. You could even decide to completely return to the original configuration and start from a known-good state.\nTo enable strace from within the systemd service unit, we need to override first the ExecStart setting. By first clearing it, the entry from the vendor-supplied will be disabled. By clearing this initial entry, we can then override it with our customized line.\nExecStart= ExecStart=/usr/bin/strace --absolute-timestamps=precision:us --daemonize --follow-forks --output=/tmp/strace.log /usr/sbin/apachectl start Can\u0026rsquo;t find the strace.log file? Check first if PrivateTmp is used, which may place the file into a subdirectory.\nExamples Let\u0026rsquo;s have look at examples with cause and more advanced troubleshooting steps.\nFailed to locate executable Apache refuses to start as it gets a Failed to locate executable followed by the name of binary and Permission denied.\nDec 13 13:17:09 test systemd[1]: Starting apache2.service - The Apache HTTP Server... Dec 13 13:17:09 test (pachectl)[28050]: apache2.service: Failed to locate executable /usr/sbin/apachectl: Permission denied Dec 13 13:17:09 test (pachectl)[28050]: apache2.service: Failed at step EXEC spawning /usr/sbin/apachectl: Permission denied Dec 13 13:17:09 test systemd[1]: apache2.service: Control process exited, code=exited, status=203/EXEC The Permission denied part of the message is giving a good hint. This may be caused by file permissions, or due to the setting ExecPaths together with NoExecPaths.\nIf this happens, check the following items:\nIs the binary that is failed really a binary? (file /usr/sbin/apachectl) Is it a symbolic link? (include that path as well) Is it a shell script? (include path, usually /bin/sh) Does the binary have the executable permission bit set? (chmod +x /path/to/file) Core-dump Let\u0026rsquo;s have a look at an example where nginx does not want to start and failing with a coredump (Failed with result \u0026lsquo;core-dump\u0026rsquo;).\n# journalctl -u nginx --since=\u0026#34;today\u0026#34; Jun 22 10:13:26 test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 22 10:13:27 test systemd[1]: nginx.service: Control process exited, code=dumped, status=31/SYS Jun 22 10:13:27 test systemd[1]: nginx.service: Failed with result \u0026#39;core-dump\u0026#39;. Jun 22 10:13:27 test systemd[1]: Failed to start A high performance web server and a reverse proxy server. The logging above does not give us a real clue why this failed. So let\u0026rsquo;s have a look at the journal, but without the filter for just the nginx service.\nJun 22 10:13:26 test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 22 10:13:26 test audit[48711]: SECCOMP auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined pid=48711 comm=\u0026#34;nginx\u0026#34; exe=\u0026#34;/usr/sbin/nginx\u0026#34; sig=31 arch=c000003e syscall=41 compat=0 ip=0x7f698563db3b code=0x80000000 Jun 22 10:13:26 test kernel: kauditd_printk_skb: 1 callbacks suppressed Jun 22 10:13:26 test kernel: audit: type=1326 audit(1719051206.884:68): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined pid=48711 comm=\u0026#34;nginx\u0026#34; exe=\u0026#34;/usr/sbin/nginx\u0026#34; sig=31 arch=c000003e syscall=41 compat=0 ip=0x7f698563db3b code=0x80000000 Jun 22 10:13:27 test systemd[1]: nginx.service: Control process exited, code=dumped, status=31/SYS Jun 22 10:13:27 test systemd[1]: nginx.service: Failed with result \u0026#39;core-dump\u0026#39;. Jun 22 10:13:27 test systemd[1]: Failed to start A high performance web server and a reverse proxy server. In this case we see SECCOMP showing an issue for our nginx process. So that probably means our process wanted to use a syscall it was not allowed to use. Fortunately, we can troubleshoot this, as the syscall (41) is mentioned. The challenge is that this is a number, which does not say much yet.\nNext step is getting this syscall number translated to the underlying name for our platform (uname -m shows \u0026lsquo;x86_64\u0026rsquo;). If we look this up online or in the kernel source , we see this is related to the syscall socket(2). Newer architectures typically use the generic syscall table .\nAs we now know that this syscall is used for creating a new socket to allow network communication on a port, we know that an important part of the networking functionality failed. As SECCOMP shows this message, it is most likely related to a syscall being blocked, or a group of syscalls. We have to check if we have the option SystemCallFilter set in our unit file.\nWe can query this property and see if it exists and is set:\n# systemctl show --no-pager --property=SystemCallFilter nginx.service SystemCallFilter=_llseek _newselect access add_key alarm arch_prctl brk cacheflush capget capset chdir chmod chown chown32 clock_getres clock_getres_time64 clock_gettime clock_gettime64 clock_nanosleep clock_nanosleep_time64 clone clone3 close close_range copy_file_range creat dup dup2 dup3 epoll_create epoll_create1 epoll_ctl epoll_ctl_old epoll_pwait epoll_pwait2 epoll_wait epoll_wait_old eventfd eventfd2 execve execveat exit exit_group faccessat faccessat2 fadvise64 fadvise64_64 fallocate fchdir fchmod fchmodat fchown fchown32 fchownat fcntl fcntl64 fdatasync fgetxattr flistxattr flock fork fremovexattr fsetxattr fstat fstat64 fstatat64 fstatfs fstatfs64 fsync ftruncate ftruncate64 futex futex_time64 futimesat get_mempolicy get_robust_list get_thread_area getcpu getcwd getdents getdents64 getegid getegid32 geteuid geteuid32 getgid getgid32 getgroups getgroups32 getitimer getpgid getpgrp getpid getppid getpriority getrandom getresgid getresgid32 getresuid getresuid32 getrlimit getrusage getsid gettid gettimeofday getuid getuid32 getxattr inotify_add_watch inotify_init inotify_init1 inotify_rm_watch io_cancel io_destroy io_getevents io_pgetevents io_pgetevents_time64 io_setup io_submit io_uring_enter io_uring_register io_uring_setup ioctl ioprio_get ioprio_set ipc kcmp keyctl kill lchown lchown32 lgetxattr link linkat listxattr llistxattr lremovexattr lseek lsetxattr lstat lstat64 madvise mbind membarrier migrate_pages mkdir mkdirat mknod mknodat mlock mlock2 mlockall mmap mmap2 move_pages mprotect mq_getsetattr mq_notify mq_open mq_timedreceive mq_timedreceive_time64 mq_timedsend mq_timedsend_time64 mq_unlink mremap msgctl msgget msgrcv msgsnd msync munlock munlockall munmap name_to_handle_at nanosleep newfstatat nice oldfstat oldlstat oldolduname oldstat olduname open openat openat2 pause personality pidfd_open pidfd_send_signal pipe pipe2 poll ppoll ppoll_time64 prctl pread64 preadv preadv2 prlimit64 process_madvise process_vm_readv process_vm_writev pselect6 pselect6_time64 pwrite64 pwritev pwritev2 read readahead readdir readlink readlinkat readv remap_file_pages removexattr rename renameat renameat2 request_key restart_syscall rmdir rseq rt_sigaction rt_sigpending rt_sigprocmask rt_sigqueueinfo rt_sigreturn rt_sigsuspend rt_sigtimedwait rt_sigtimedwait_time64 rt_tgsigqueueinfo sched_get_priority_max sched_get_priority_min sched_getaffinity sched_getattr sched_getparam sched_getscheduler sched_rr_get_interval sched_rr_get_interval_time64 sched_setaffinity sched_setattr sched_setparam sched_setscheduler sched_yield select semctl semget semop semtimedop semtimedop_time64 sendfile sendfile64 set_mempolicy set_robust_list set_thread_area set_tid_address set_tls setfsgid setfsgid32 setfsuid setfsuid32 setgid setgid32 setgroups setgroups32 setitimer setns setpgid setpriority setregid setregid32 setresgid setresgid32 setresuid setresuid32 setreuid setreuid32 setrlimit setsid setuid setuid32 setxattr shmat shmctl shmdt shmget sigaction sigaltstack signal signalfd signalfd4 sigpending sigprocmask sigreturn sigsuspend splice stat stat64 statfs statfs64 statx swapcontext symlink symlinkat sync sync_file_range sync_file_range2 syncfs sysinfo tee tgkill time timer_create timer_delete timer_getoverrun timer_gettime timer_gettime64 timer_settime timer_settime64 timerfd_create timerfd_gettime timerfd_gettime64 timerfd_settime timerfd_settime64 times tkill truncate truncate64 ugetrlimit umask uname unlink unlinkat unshare userfaultfd utime utimensat utimensat_time64 utimes vfork vmsplice wait4 waitid waitpid write writev So yes, this is definitely being active. This list will show the allowed syscalls, so let\u0026rsquo;s have a look.\nAfter a looking at the sorted list there is no \u0026lsquo;socket\u0026rsquo; available, nor is bind(2) or connect(2). As software that relies on network functions like that, it makes sense that nginx will not be able to run properly. To resolve this, allow the @network-io group in the SystemCallFilter. How do we know this is the related group? That is easy to figure out from the syscall filter overview.\n","permalink":"https://linux-audit.com/systemd/troubleshooting-a-failed-systemd-unit/","tags":["linux","systemd","troubleshooting"],"title":"Troubleshooting a failed systemd unit (with examples)"},{"categories":["System Administration"],"contents":"Systemd stores the configuration for units, like services, in individual unit files. When changes are made to these units, a reload might be needed. This is where systemctl daemon-reload comes into play. But what exactly does the daemon-reload subcommand really do? In short: rerun generators, reload units files, recreate the dependency tree. Let\u0026rsquo;s have a look at the more detailed answer.\nRunning generators Generators are helper scripts to convert non-native scripts to unit files that are usable by systemd. These generators are used during the boot, but also when the daemon configuration is reloaded. In multiple phases all generators are started in parallel, with the goal to minimize the time tasks are waiting for each other to finish.\nReloading the units The configuration of the units needs to be checked and reloaded. Any change that has a different condition than before, might trigger a reload or restart of a unit.\nDependency tree update Systemd is focused on optimizing how units interact with each other, like which service depends on another service or unit. This set of dependencies is updated, so services are started in the right order.\n","permalink":"https://linux-audit.com/systemd/faq/what-does-systemctl-daemon-reload-do/","tags":["faq","linux","systemctl","systemd"],"title":"What does systemctl daemon-reload do?"},{"categories":["System Administration"],"contents":"Systemd may need to reload a part of the unit configuration if changes were made. To find out if the related systemctl daemon-reload command is needed, the state of the individual units can be tested. This is done by querying the property using the --property=NeedDaemonReload option.\nTesting a single service like nginx, can be done this way:\n# systemctl show --property=NeedDaemonReload --value nginx.service yes This output will return a \u0026lsquo;yes\u0026rsquo; or \u0026rsquo;no\u0026rsquo; value.\nCheck all services In need for a one-liner to check if there is any service that was changed and requires systemctl to initiate the daemon-reload?\n# for UNIT in $(systemctl list-units --type=service --all --legend=false --plain | awk \u0026#39;{print $1}\u0026#39;); do if [ \u0026#34;$(systemctl show --property=NeedDaemonReload --value \u0026#34;${UNIT}\u0026#34;)\u0026#34; = \u0026#34;yes\u0026#34; ]; then echo \u0026#34;$UNIT is outdated, systemctl daemon-reload needed\u0026#34;; fi; done nginx.service is outdated, systemctl daemon-reload needed ","permalink":"https://linux-audit.com/systemd/faq/how-to-check-if-systemctl-daemon-reload-is-needed/","tags":["faq","linux","systemctl","systemd"],"title":"How to check if 'systemctl daemon-reload' is needed"},{"categories":["System Administration"],"contents":"Systemd can restrict services from using particular syscalls with the help of the unit setting SystemCallFilter. Instead of mentioning all individual syscalls, systemd has predefined sets that can be used. These sets group functions that are related. To see which syscalls are part of a set, use the systemd-analyze command.\n# systemd-analyze syscall-filter @ipc @ipc # SysV IPC, POSIX Message Queues or other IPC ipc memfd_create mq_getsetattr mq_notify mq_open mq_timedreceive mq_timedreceive_time64 mq_timedsend mq_timedsend_time64 mq_unlink msgctl msgget msgrcv msgsnd pipe pipe2 process_madvise process_vm_readv process_vm_writev semctl semget semop semtimedop semtimedop_time64 shmat shmctl shmdt shmget See systemd syscall filtering for all details.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-which-syscalls-are-part-of-systemd-syscall-filter-set/","tags":["faq","linux","syscall","systemd-analyze","systemd"],"title":"How to see which syscalls are part of a systemd syscall filter set"},{"categories":null,"contents":"Why and when to use SystemCallFilter Systemd has the unit setting SystemCallFilter which aims to prevent misuse of syscalls that are not needed for normal functioning of a process or its children. This powerful filtering uses seccomp and defines what syscalls are allowed. In other words, it restricts the abilities of a process (sandboxing). At the same time, it requires understanding of processes by the system administrator. See the overview of Linux syscalls for more details.\nConfiguration This setting takes a space-separated list and may be specified multiple times.\nAllow-listing By default the list contains the entries of allowed system call names. Unspecified syscalls will be denied and result in its process execution being stopped immediately.\nDeny-listing Another approach is to reverse this action by using a ~ as its first character. The principle of deny-listing will then be used. A match then will stop the execution of the process.\nCombination When both (allow-listing and deny-listing) are used, the first match will take precedence and define the default action to take.\nFilter sets Instead of defining each syscall, there are also predefined sets that can be used:\n@aio @basic-io @chown @clock @cpu-emulation @debug @file-system @io-event @ipc @keyring @memlock @module @mount @network-io @obsolete @pkey @privileged @process @raw-io @reboot @resources @sandbox @setuid @signal @swap @sync @system-service @timer @known See systemd syscall filtering for more details, usage, and information about the predefined sets.\nTo know what syscalls are part of a set:\n# systemd-analyze syscall-filter @mount @mount # Mounting and unmounting of file systems chroot fsconfig fsmount fsopen fspick mount mount_setattr move_mount open_tree pivot_root umount umount2 Generic advice This setting is a powerful option to restrict what processes can do.\n","permalink":"https://linux-audit.com/systemd/settings/units/systemcallfilter/","tags":["configuration","linux","mount","sandboxing","seccomp","syscall","systemd"],"title":"SystemCallFilter setting"},{"categories":["System Administration"],"contents":"Introduction Systemd uses seccomp to implement filtering by syscalls. Syscalls are system functions and are usually provided by GLIBC. This is a generic library full with functions to allow communication between a process and the kernel. With seccomp support, these syscalls can be blocked. Systemd uses this to allow or deny specific systems functions with the SystemCallFilter.\nBesides allowing or deny specific syscalls, systemd also provides predefined sets. These sets group similar or related functionality into a filter set, that then can be used to allow or deny.\nWhat syscalls does a process use? A web server running nginx should obviously be allowed to listen to network traffic for a port like 80 or 443. Most likely it does not need to be able to change the system clock, while a NTP daemon should. But how do you know what syscalls are used in the first place?\nDynamic analysis Want to discover functions what syscalls are used by a running process? You may use strace on a running process, although this may crash it or decrease its performance. So when possible do this only on systems that are not in production.\nBinary analysis Another way is looking at the binary and see what functions are used.\nstrings /usr/sbin/nginx | grep -E '^[a-z0-9_]{4,32}\\(\\)' | awk '{print $1}' | sort | uniq\nFilter sets Systemd uses filter sets to allow or deny functionality per group. To see the content of a set (e.g. @clock):\nsystemd-analyze syscall-filter @clock\nTo simplify looking up this information, they are collected here in this overview.\n@default Description: These system calls are always permitted\nSyscall Purpose arch_prctl brk Change the location of program break, specifically the end of the process\u0026#39;s data segment cacheflush Flushes contents of cache(s) for user addresses in specified range clock_getres Retrieve the resolution (precision) of a specified clock clock_getres_time64 clock_gettime Retrieve time from specified clock clock_gettime64 64-bit version of clock_gettime() clock_nanosleep clock_nanosleep_time64 execve Executes the program referred to by specified pathname exit Terminates the calling process, parent process will receive a SIGCHLD signal exit_group futex Provides a method for waiting until certain condition becomes true futex_time64 futex_waitv get_robust_list get_thread_area getegid Returns effective group ID of the calling process getegid32 geteuid Retrieve effective user ID of calling process geteuid32 32-bit version of geteuid() getgid Returns real group ID of the calling process getgid32 32-bit version of getgid() getgroups Returns supplementary group IDs of calling process getgroups32 32-bit version of getgroups() getpgid Retrieve process group ID (PGID) getpgrp Retrieves process group ID (PGID) of the calling process getpid Returns process ID (PID) of calling process getppid Returns process ID (PID) of parent of the calling process getrandom Receive random bytes getresgid getresgid32 getresuid getresuid32 getrlimit Get resource limits getsid Receive session ID of a defined process gettid Returns thread ID (TID) of caller. Same as process ID (PID) for single-threaded applications, otherwise different gettimeofday Get time or timezone getuid Get resource limits getuid32 32-bit version of getuid() membarrier mmap Create new mapping in the virtual address space of the calling process mmap2 mprotect Sets protection on a defined region of memory munmap Deletes the mappings for specified address range and marks range to generate invalid memory references nanosleep Suspends the execution of the thread calling the request pause Changes the thread or process that is calling the request to sleep until the moment a signal is received prlimit64 restart_syscall riscv_flush_icache riscv_hwprobe rseq rt_sigreturn sched_getaffinity Retrieves the mask on which CPUs the process thread can run sched_yield Request by the calling thread to free up itself from the CPU and move it to the very end of the queue, so the next thread can run set_robust_list set_thread_area set_tid_address set_tls sigreturn time Return time; as number of seconds since the Epoch (1970-01-01 00:00:00 \u0026#43;0000 (UTC)) ugetrlimit @aio Description: Asynchronous IO\nSyscall Purpose io_cancel Attempts to cancel asynchronous I/O operation that was submitted by io_submit() io_destroy io_getevents io_pgetevents io_pgetevents_time64 io_setup io_submit Submit asynchronous I/O blocks for processing, can be cancelled with io_cancel() io_uring_enter io_uring_register io_uring_setup @basic-io Description: Basic IO\nAlmost all software requires this set to open a file, read from it, or write to it.\nSyscall Purpose _llseek close Close file descriptor close_range Close the file descriptors of the selected range dup Duplicate file descriptor; more specifically it allocates a new file descriptor that also refers to open file description oldfd dup2 Same as dup(), duplicate file descriptor; difference is that it uses file descriptor number specified in newfd dup3 Same as dup2(); difference is that caller can force close-on-exec flag (O_CLOEXEC) to be set lseek Reposition file offset for read/write pread64 Similar to pread() but with an offtype of type off64_t that allows changing file positions in files larger than two gigabytes preadv Reads data into multiple buffers as readv(), with option to set an offset where in the file the read operation is to be performed. preadv2 Reads data into multiple buffers like preadv(), with additional flags pwrite64 Similar to pwrite() but with an offtype of type off64_t that allows changing file positions in files larger than two gigabytes pwritev Write to file description like writev(), except that multiple buffers are written pwritev2 Write to file description like pwritev() with multiple buffers, additionally has extra flags read Read from file descriptor readv Read buffers from file write Write to file descriptor writev Writes buffers to file @chown Description: Ability to change ownership of files and directories\nSyscall Purpose chown Changes ownership of file specified by pathname, dereferenced if file is a symbolic link chown32 fchown Changes ownership of file, referred to by open file descriptor (fd) fchown32 fchownat Similar to fchown(), but deals differently with relative paths lchown Like chown(), does not dereference symbolic links lchown32 @clock Description: Ability to change system time\nNote: this is rarely needed for normal services.\nSyscall Purpose adjtimex Reads and optionally sets adjustment parameters for clock adjustment algorithm used on Linux (RFC 5905) clock_adjtime Reads and optionally sets adjustment parameters for clock adjustment algorithm used on Linux (RFC 5905). It behaves like adjtimex(), but takes an additional clk_id argument to define the clock clock_adjtime64 clock_settime Set time of specified clock clock_settime64 settimeofday Set time or timezone @cpu-emulation Description: Ability to do CPU emulation\nSyscall Purpose modify_ldt subpage_prot switch_endian vm86 vm86old @debug Description: Debugging, performance monitoring, tracing functionality\nNote: this is normally only used by tools like strace and perf.\nSyscall Purpose lookup_dcookie perf_event_open pidfd_getfd ptrace Process tracing; usually for breakpoint debugging and system call tracing rtas s390_runtime_instr sys_debug_setcontext @file-system Description: File system operations\nNote: normally all processes need this to be able to read a directory or open a file.\nSyscall Purpose access Checks whether the calling process can access the pathname, dereferenced when it is a symbolic link chdir Change work directory chmod Change mode of the file, dereferenced for symbolic links close Close file descriptor creat Like open(), but sets flags O_CREAT|O_WRONLY|O_TRUNC faccessat Similar to access(), works slightly different when pathname is relative faccessat2 Closely similar to faccessat() but implements flags argument to correct incorrect implementation in faccessat() fallocate Allows the caller to manipulate allocated disk space for a file fchdir Similar to chdir, but uses open file descriptor fchmod Same as chmod, but used file by open file descriptor fd fchmodat Similar to chmod(), works slightly different when pathname is relative fchmodat2 fcntl Performs an action on file defined by a file descriptor, such as setting flags fcntl64 64-bit version of fcntl64() fgetxattr Retrieves xattr (extended attributes) by using a file descriptor returned by open() flistxattr Like listxattr() it retrieves the list of extended attributes (xattr) but associated with an open file descriptor fremovexattr fsetxattr fstat Similar to stat(), but uses file descriptor fd fstat64 fstatat64 fstatfs Returns information about an open file fstatfs64 ftruncate Truncate a file open for writing to specified number of bytes, which may fill it with null bytes (\\0) or decrease its size and losing data ftruncate64 64-bit version of ftruncate() futimesat getcwd Copies the absolute pathname of current working directory to a buffer getdents Retrieve entries from a directory getdents64 64-bit version of getdents() getxattr Retrieves xattr (extended attributes) from a given file defined by its path and name inotify_add_watch Adds a new watch or change an existing watch for a file inotify_init Initializes new inotify instance and returns file descriptor that is associated with a new inotify event queue inotify_init1 Like inotify_init() it initializes a new inotify instance and returns file descriptor that is associated with a new inotify event queue, with additional flags inotify_rm_watch Removes a watch with a watch descriptor (wd) from an inotify instance specified by its file descriptor (fd) lgetxattr Retrieves xattr (extended attributes) for symbolic links itself, not the destination it links to link Create new link (hard link) to existing file linkat Similar to link(), but deals differently with relative paths listxattr Retrieves the list of extended attributes (xattr) associated with the path llistxattr Like listxattr, it retrieves the list of extended attributes (xattr) associated with the path, but if it is a symbolic link it will give information about the link, not the destination target lremovexattr lsetxattr lstat Similar to stat(), but if pathname is symbolic link, return information about link and not the file that symbolic link points to lstat64 mkdir Create directory mkdirat Similar to mkdir() but deals differently with relative paths mknod Create filesystem node (file, device special file, or named pipe) named pathname mknodat Similar to mknod, works slightly different when pathname is relative newfstatat oldfstat oldlstat oldstat open Opens file specified by pathname to allow reading or writing data openat Similar to open(), but uses dirfd argument and deals differently with path openat2 readlink Places a copy of the symbolic link referenced by its path into a buffer readlinkat Like readlinkat() it places a copy of the symbolic link referenced by its path into a buffer, but can use relative path removexattr rename Rename a file, move it between directories if required renameat Similar to rename(), with deals differently with relative paths renameat2 Similar to renameat() when no flags are provided, otherwise it has additional options rmdir Delete directory setxattr stat Get information about file stat64 64-bit version of stat() statfs statfs64 statx symlink Create symbolic link symlinkat Similar to symlink() but deals differently with relative paths truncate Truncate a writable file to specified number of bytes, which may fill it with null bytes (\\0) or decrease its size and losing data truncate64 64-bit of truncate() unlink Delete name from filesystem unlinkat Similar to unlink() but deals differently with relative paths utime Change access and modification times of inode utimensat utimensat_time64 utimes Similar to utime(), but uses array instead of a structure @io-event Description: Event loop system calls\nSyscall Purpose _newselect epoll_create epoll_create1 epoll_ctl Manage (add, modify, remove) entries in epoll instance, which is used to monitor if I/O is allowed on the defined set of file descriptors. Similar to poll(), with additional benefits. epoll_ctl_old epoll_pwait epoll_pwait2 epoll_wait Waits for events on an epoll instance which is defined by a file descriptor (epfd) epoll_wait_old eventfd eventfd2 poll Similar task to select(2), which is waiting for a set of file descriptors to become available for I/O. ppoll Let an application wait until file descriptor is available or signal is caught ppoll_time64 pselect6 pselect6_time64 select Let a program monitor multiple file descriptors until one or more become available for I/O actions. This system call has limitations and typically poll or epoll is used. @ipc SysV IPC, POSIX Message Queues or other Inter-Process Communication (IPC)\nSyscall Purpose ipc memfd_create mq_getsetattr mq_notify mq_open mq_timedreceive mq_timedreceive_time64 mq_timedsend mq_timedsend_time64 mq_unlink msgctl msgget msgrcv msgsnd pipe Create a pipe that allows unidirectional communication between processes pipe2 Similar to pipe(), to create a channel between two processes. With flag O_DIRECT it will use packet-style communication instead of a stream process_madvise process_vm_readv process_vm_writev semctl semget semop semtimedop semtimedop_time64 shmat shmctl shmdt shmget @keyring Kernel keyring access\nSyscall Purpose add_key Create or update a key for kernel key management facility keyctl Allow user-space programs to take actions on keys, such as updating, revocation, ownership request_key Request a key from kernel key management facility @memlock Memory locking control\nSyscall Purpose mlock Lock pages in a specified address range, so they are guaranteed to stay in memory instead of being swapped to disk mlock2 Same as mlock() if flags is 0. With flag MLOCK_ONFAULT is locks the current resident pages, the mark the range so currently nonresident pages are locked later when they are used (page fault) mlockall Similar to mlock, but tries to lock all the memory pages of the calling process to prevent swapping munlock Opposite of mlock() to release lock on memory area, so it can be swapped to disk if needed munlockall Unlocks all memory pages of calling process so it can be swapped to disk again by the kernel @module Description: Ability to load or unload kernel modules\nSyscall Purpose delete_module Tries to remove an unused loadable module entry which is related currently loaded Linux kernel module (LKM) finit_module Similar to init_module(); loads image (ELF) but refers to a file description init_module Load image (ELF) into the kernel space including the required steps to initialize it, including triggering the init() function of the module @mount Description: Ability to mount or unmount a file system\nNote: Most services will not need to use mount/umount\nSyscall Purpose chroot The root directory (which is normally /) of the calling process will be changed to the one specified in the path to sandbox a process fsconfig fsmount fsopen fspick mount mount_setattr move_mount open_tree pivot_root umount umount2 @network-io Description: Network or Unix socket actions, like opening a network port to listen\nWhen to use: This filter set is only required for services that actually listen to a socket on the network.\nSyscall Purpose accept Accept a connection on a socket accept4 bind Assigns address to a socket that was created with socket() connect Initiate connection on a defined socket getpeername Receive address of the peer connected to a socket getsockname Retrieve current address of defined socket getsockopt Get options for socket listen Marks socket as a passive to allow it accepting incoming connections with accept() recv Like read(), but normally only used on a socket and has additional flags that can be set recvfrom Receives a message on a socket, close to recv(), but with additional flags related to receiving source recvmmsg recvmmsg_time64 recvmsg Receives a message on a socket with a predefined structure to minimize the number of arguments send sendmmsg sendmsg sendto setsockopt Set options on socket shutdown socket Create endpoint for communication and return file descriptor socketcall socketpair Create a pair of connected sockets, for example for communication between parent and child process @obsolete Description: Unusual, obsolete or unimplemented system calls, with some unknown to the underlying seccomp library\nSyscall Purpose _sysctl afs_syscall bdflush break create_module ftime get_kernel_syms getpmsg gtty idle lock mpx prof profil putpmsg query_module security sgetmask ssetmask stime stty sysfs tuxcall ulimit uselib ustat vserver @pkey Description: Set of calls for memory protection keys\nSyscall Purpose pkey_alloc pkey_free pkey_mprotect @privileged Description: System calls which typically need super-user capabilities. It includes also other filter sets:\n@chown @clock @module @raw-io @reboot @swap Syscall Purpose _sysctl acct bpf capset chroot The root directory (which is normally /) of the calling process will be changed to the one specified in the path to sandbox a process fanotify_init fanotify_mark nfsservctl open_by_handle_at pivot_root quotactl quotactl_fd setdomainname Sets the NIS domain name to the defined value setfsuid setfsuid32 setgroups setgroups32 sethostname Sets the hostname to the defined value setresuid setresuid32 setreuid setreuid32 setuid setuid32 vhangup @process Description: Process control, execution, namespacing operations\nSyscall Description capget retrieve thread capabilities clone similar to fork() to create a child process, with more fine-grained options to define what is shared between calling process and child. This system call can also make a new process part of newly created namespace by specifying a flag. clone3 provides superset of the functionality of the older clone() interface to create child process execveat fork create a new child process by duplicating the calling process, with caller becoming the parent process getrusage kill pidfd_open obtain a file descriptor referring to a process pidfd_send_signal prctl Perform operations on a process or thread, such as changing its capabilities, set name of the calling thread, set the secure computing mode (seccomp), and more. rt_sigqueueinfo rt_tgsigqueueinfo setns allows calling thread to switch to a different namespace swapcontext tgkill times get process and child process times, including CPU time in userspace and by the system for the calling process, and similar for the child processes tkill unshare allows a process to unshare parts of its execution context, such as mount namespace, from other processes. Parts of the execution context are automatically shared with other processes when fork(2), vfork(2) or clone(2) are used. With this syscall it does not have not to create a new process. vfork wait4 waitid waitpid Suspend the execution of the calling process thread until one of the specified child processes (by PID) terminates. By using specific options, also other actions like termination can be waited for. The functionality of this system call is similar to wait(), yet with more control over the children and states. @raw-io Description: raw I/O port access\nSyscall Purpose ioperm iopl pciconfig_iobase pciconfig_read pciconfig_write s390_pci_mmio_read s390_pci_mmio_write @reboot Description: ability to reboot or reboot preparation using kexec functionality that loads the kernel for later execution.\nNote: normal services do not need this set of syscalls\nSyscall Purpose kexec_file_load Similar to kexec_load(), but uses file descriptor for kernel and initrd (initial ram disk) kexec_load Load new kernel for later execution reboot Reboots the system, or enables/disables reboot keystroke (default: Ctrl\u0026#43;Alt\u0026#43;Delete; changed using loadkeys(1)) @resources Description: ability to alter resource settings, such as process priority\nSyscall Purpose ioprio_set mbind migrate_pages move_pages nice Change process priority, with \u0026#43;19 (lowest priority) up to to -20 (high priority) sched_setaffinity Defines by using a mask on which CPUs the process thread can run sched_setattr sched_setparam sched_setscheduler set_mempolicy set_mempolicy_home_node setpriority setrlimit Set resource limits @sandbox Description: sandbox functionality, such as support for landlock and seccomp\nSyscall Purpose landlock_add_rule landlock_create_ruleset landlock_restrict_self seccomp @setuid Description: Operations to changing user/group credentials (setuid/setgid)\nSyscall Purpose setgid Set effective group ID of calling process, with CAP_SETGID capability it also sets real GID and saved set-group-ID setgid32 setgroups setgroups32 setregid setregid32 setresgid setresgid32 setresuid setresuid32 setreuid setreuid32 setuid Set effective user ID of calling process, with CAP_SETUID capability it also sets real UID and saved set-user-ID setuid32 @signal Description: signal handling for processes\nSyscall Purpose rt_sigaction rt_sigpending rt_sigprocmask rt_sigsuspend rt_sigtimedwait rt_sigtimedwait_time64 sigaction sigaltstack signal signalfd signalfd4 sigpending sigprocmask sigsuspend @swap Description: ability to enable or disable swap devices\nNote: not required for normal services\nSyscall Purpose swapoff swapon @sync Description: synchronize files and memory to storage\nSyscall Purpose fdatasync fsync msync sync sync_file_range sync_file_range2 syncfs @system-service General system service operations\nBesides the syscalls below, it also includes the following filter sets:\n@aio @basic-io @chown @default @file-system @io-event @ipc @keyring @memlock @network-io @process @resources @setuid @signal @sync @timer Syscall Purpose arm_fadvise64_64 capget Retrieve thread capabilities capset Set thread capabilities copy_file_range fadvise64 fadvise64_64 flock Apply or remove advisory lock on file get_mempolicy getcpu getpriority ioctl ioprio_get kcmp madvise mremap name_to_handle_at oldolduname olduname personality readahead readdir Read a directory remap_file_pages sched_get_priority_max sched_get_priority_min sched_getattr sched_getparam sched_getscheduler sched_rr_get_interval sched_rr_get_interval_time64 sched_yield Request by the calling thread to free up itself from the CPU and move it to the very end of the queue, so the next thread can run sendfile Copies data between one file descriptor and another sendfile64 setfsgid Sets the group identity used for performing file system checks setfsgid32 setfsuid setfsuid32 setpgid Sets the process group ID (PGID) of the process setsid splice sysinfo tee Duplicate pipe content, does not consume the data umask Set file mode creation mask uname Retrieve name and information about the current kernel userfaultfd vmsplice @timer Description: Timers, to schedule operations by time\nSyscall Description alarm schedule an alarm; it lets the system generate a SIGALRM signal for the process after a specified time getitimer setitimer timer_create timer_delete timer_getoverrun timer_gettime timer_gettime64 timer_settime timer_settime64 timerfd_create timerfd_gettime timerfd_gettime64 timerfd_settime timerfd_settime64 times get process and child process times, including CPU time in userspace and by the system for the calling process, and similar for the child processes @known Description: Includes all syscalls that are known to the Linux kernel, plus the ones in @obsolete\n","permalink":"https://linux-audit.com/systemd/systemd-syscall-filtering/","tags":["awk","bpf","linux","namespaces","seccomp","security","syscall","systemd"],"title":"Systemd syscall filtering"},{"categories":["System Administration"],"contents":"Systemd and its services can be in several states, such as enabled, disabled, failed, running. If you no longer need a particular service to run, then the first step is to stop a service.\nsystemctl stop nginx.service\nBut stopping a service is not the same as disabling a service. With that comes a very frequently asked question: what is the difference between a service that is disabled and one that is masked?\nDifference disable and mask When using disable to disabling a service, the underlying symlinks are removed. These symlinks normally activate the service during the boot process. Disabling a service sounds like it is not allowed to run, but it still may be activated if it is a dependency of another service.\nUsing mask will actually create a symlink pointing to /dev/null . This will making it no longer possible to start the service, until the moment that the unmask subcommand is used.\nSo the differences between disable and mask is that the latter is the most forceful way of disabling a unit.\n","permalink":"https://linux-audit.com/systemd/faq/what-is-the-difference-between-systemctl-disable-and-systemctl-mask/","tags":["faq","linux","systemctl","systemd"],"title":"What is the difference between systemctl disable and systemctl mask?"},{"categories":["System Administration"],"contents":"Introduction into changing systemd units Systemd allows service units to be changed and customized. This is done using a drop-in file, which is often called override.conf. It overrides the vendor-supplied version of a service to customize it. Instead of duplicating the configuration, the override file contains the differences.\nEditing a service file Changing a service can be done using systemctl, followed by the edit subcommand and the service unit.\nsystemctl edit myservice.service\nWhen using this command, it will open your default editor, such as vi. At this stage you can make any changes to the service file. Any changes should be placed between the comment section at the top and the comment section a little bit lower. Do not remove these comment sections, as systemctl uses these to see what changes you made.\nEditing the primary unit To edit the main configuration file, use the --full option. This is typically not a good idea for services that were installed using a package manager, but it is for your custom service units.\nsystemctl edit --full myservice.service\nOverriding list entries Some properties of a service may be repeated. These lists may for example define a path. It might be needed to first reset that property by defining it in your override file with an empty value. After that entry, add them in. If you also need the values from the original main configuration, add those as well.\nConfiguration check After the edit action, ensure that the configuration has the right entries.\nsystemctl cat myservice.service\nActivating changes When making changes to units, a reload of systemd might be needed.\nsystemctl daemon-reload\nThen reload the service.\nsystemctl restart myservice.service\n","permalink":"https://linux-audit.com/systemd/faq/how-to-use-systemctl-edit/","tags":["faq","linux","systemctl","systemd"],"title":"How to use systemctl edit to change a service?"},{"categories":["System Administration"],"contents":"The systemctl command will normally all active units. To filter this output to just the running services, we can combine the options --type= and --state=. For this particular case we set the type to service and the type state to running.\nUsage # systemctl --type=service --state=running --legend=false accounts-daemon.service loaded active running Accounts Service avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack colord.service loaded active running Manage, Install and Generate Color Profiles dbus-broker.service loaded active running D-Bus System Message Bus firewalld.service loaded active running firewalld - dynamic firewall daemon gdm.service loaded active running GNOME Display Manager NetworkManager.service loaded active running Network Manager polkit.service loaded active running Authorization Manager qemu-guest-agent.service loaded active running QEMU Guest Agent rtkit-daemon.service loaded active running RealtimeKit Scheduling Policy Service spice-vdagentd.service loaded active running Agent daemon for Spice guests sshd.service loaded active running OpenSSH Daemon systemd-journald.service loaded active running Journal Service systemd-logind.service loaded active running User Login Management systemd-timesyncd.service loaded active running Network Time Synchronization systemd-udevd.service loaded active running Rule-based Manager for Device Events and Files systemd-userdbd.service loaded active running User Database Manager udisks2.service loaded active running Disk Manager upower.service loaded active running Daemon for power management user@1000.service loaded active running User Manager for UID 1000 wpa_supplicant.service loaded active running WPA supplicant This command will show just services that are running. The legend is set to false, meaning no header and footer are displayed.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-only-running-services-with-systemctl/","tags":["faq","linux","systemctl","systemd"],"title":"How to see only running services with systemctl"},{"categories":["System Administration"],"contents":"Many Linux distributions are using systemd as its service manager. In release 256 the new tool run0 was introduced. In this cheat sheet commands are collected to quickly use the capabilities of run0.\nNever used run0 before? Have a look at run0 introduction and usage.\nMissing something? Let it know!\nCommonly used options Option Action --background=[COLOR] Set background color, or disable when set empty value --chdir=PATH Set current working directory to PATH --description=\u0026quot;TEXT\u0026quot; Give the transient unit a custom description --nice=VALUE Define nice level (19 to -20) --property=NAME=VALUE Set property (e.g. sandboxing/resource limitation) --setenv=ENV=VALUE Declare an environment variable ENV with value VALUE --unit=NAME Define a name of our transient unit instead of random one Basic usage To elevate permissions without running a specific command, run run0 without any parameters.\nrun0\nSet unit name and description run0 --unit=mynewunit --description=\u0026quot;This is a new unit\u0026quot; systemctl status mynewunit.service\nBackground color Use a blue background instead of red.\nrun0 --background=\u0026quot;44\u0026quot; ps -ef\nDisable color:\nrun0 --background= ps -ef\nNice level run --nice=19 my-task-with-low-priority\nEnvironment variable run --setenv=SECRET=true bash -c 'export'\nSet a property Define a property to apply sandboxing or restrict system resources.\nrun0 --property=ProtectSystem=strict bash -c 'echo test \u0026gt; /var/log/this-will-fail'\n","permalink":"https://linux-audit.com/cheat-sheets/run0/","tags":["cheatsheet","howto","linux","run0","systemd"],"title":"run0 cheat sheet"},{"categories":["System Administration"],"contents":"Introduction Run0 was introduced in systemd version 256 and is intended as an alternative to sudo. Both commands elevate privileges, but are also somewhat different. Author Lennart Poettering describes run0 as somewhat more similar to the ssh command than it is to sudo.\nBasics When you use run0, it will create a transient service unit. Opposed to a permanent service unit that will run for a long time, transient units typically run very shortly. It can be compared with a towel versus a disposable paper towel. When the task is completed or you switched back to an unprivileged user, the service unit will be gone.\nDifferences with sudo Run0 uses isolated services (transient units). Almost no execution or security context credentials are transferred into this new service. Run0 allocates a pseudo-tty, further isolating the unit. When possible, the authentication prompt is isolated from the terminal. This is done via polkit authentication. SetUID and SetGID file access bit are not used. No configuration file like /etc/sudoers. Usage and basic options Like most tools, run0 comes with a range of options. Here are some of the basic ones that you probably will use.\nOption Action --background=[COLOR] Set background color, or disable when set empty value --chdir=PATH Set current working directory to PATH --description=\u0026quot;TEXT\u0026quot; Give the transient unit a custom description --nice=VALUE Define nice level (19 to -20) --property=NAME=VALUE Set property (e.g. sandboxing/resource limitation) --setenv=ENV=VALUE Declare an environment variable ENV with value VALUE --unit=NAME Define a name of our transient unit instead of random one Let\u0026rsquo;s have a look at how to use run0 and apply these options.\nBecome root user To elevate permissions without running a specific command, run run0 without any parameters.\nrun0\nBackground color You may notice that the background color changed, one of the features of run0. Don\u0026rsquo;t like this color? Let\u0026rsquo;s use a blue background instead of red.\nrun0 --background=44 ps -ef\nUse a blue background with bold characters. Use quotes when using the semicolon character.\nrun0 --background=\u0026quot;44;1\u0026quot; ps -ef\nNot interested in a background color at all? Give it an empty value.\nrun0 --background= ps -ef\nUnit and description When we run a command with run0, it will be assigned a new service unit. This is called an transient unit, one of a short duration. The name that is assigned is somewhat random, something like \u0026lsquo;run-u302\u0026rsquo;. By using the option --unit= we can give it a custom name. Additionally, we can provide a description as well.\nBy defining a clear name to the unit, it can easily be monitored. Let\u0026rsquo;s fire up a new task and look at the status of our new service unit.\n$ run0 --nice=19 --unit=slowcopy --description=\u0026#34;This is a copy task\u0026#34; systemctl status slowcopy.service ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ==== Authentication is required to manage system services or other units. Authenticating as: Michael (michael) Password: ==== AUTHENTICATION COMPLETE ==== ● slowcopy.service - This is a copy task Loaded: loaded (/run/systemd/transient/slowcopy.service; transient) Transient: yes Active: active (running) since Wed 2024-06-19 10:33:39 CEST; 4ms ago Invocation: 1e903a4c842c453dba816188f65ca8d6 Main PID: 4804 (systemctl) Tasks: 2 (limit: 4671) Memory: 1.5M (peak: 1.5M) CPU: 6ms CGroup: /user.slice/slowcopy.service ├─4804 /usr/bin/systemctl status slowcopy.service └─4805 less Change work directory The work directory defaults to the home directory of the user. To adjust this, use --chdir= followed by the work directory.\nrun0 --chdir=/var pwd\nNice level Run0 allows to run a task with elevated privileges and directly an assignment of the nice level. This is very useful for longer running tasks like a file copy. This way it won\u0026rsquo;t impact the production system too much.\nrun0 --nice=19 my-task-with-low-priority\nEnvironment variables If you need to provide an environment variable, then this can be done using --setenv= followed by the variable and its value. To test that it is really available, let\u0026rsquo;s display the environment variables from within the newly created service.\nrun0 --setenv=SECRET=true bash -c 'export'\nSandboxing capabilities with properties One of the most powerful features with run0 is to set a property. A property defines one or more aspects of the environment that processes are running in. To see available properties, use the show subcommand of systemctl.\nsystemctl show dmesg.service\nDefining a property within run0 is an interesting combination. So even though you elevate privileges, you can still define restrictions and sandbox the command or application.\nExample Let\u0026rsquo;s say you created a new shell script to automate an important system task. While it may need root privileges to run and interact with other commands, the script should not be able to wreck the system. One option is to use the ProtectSystem setting, which can mark the file system as read-only.\nLet\u0026rsquo;s run a simple task to write a string of text into a file.\n$ run0 --property=ProtectSystem=strict bash -c \u0026#39;echo test \u0026gt; /var/log/write-test\u0026#39; /usr/bin/bash: line 1: /var/log/write-test: Read-only file system So even though we run0 will give us root permissions, we are not allowed to write to a new file in our /var/log directory. But what if exactly that directory is what our script requires as part of its tasks? No problem, just set a second property that grants us write permission for that particular location.\nrun0 --property=ProtectSystem=strict --property=ReadWritePaths=/var/log/write-test bash -c 'echo test \u0026gt; /var/log/write-test'\nNow our file can be created, exactly as we intended. Testing a new script that requires root permissions has become a lot safer!\n","permalink":"https://linux-audit.com/systemd/run0-introduction-and-usage/","tags":["linux","run0","systemd"],"title":"Run0: introduction and usage"},{"categories":["System Administration"],"contents":"Systemd introduced run0 as its alternative to sudo. One of the features if a colored background when your privileges are elevated.\nTo disable this behaviour, use the option --background= with an empty value.\nrun0 --background=\nThe red background now will be gone, which can be useful if the color conflicts with the output or when it is unwanted.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-disable-run0-background-color/","tags":["faq","linux","run0","sudo","systemctl","systemd"],"title":"How to disable the background color of run0"},{"categories":null,"contents":"Why and when to use MemoryDenyWriteExecute The setting MemoryDenyWriteExecute will block the creation or alteration of a memory segment to become writable and executable as well. By enabling this limitation, it will increase the bar software exploits to change running code dynamically.\nUsage [Service] MemoryDenyWriteExecute=yes InaccessiblePaths=/dev/shm SystemCallFilter=~memfd_create Caveats To prevent circumvention of this setting, access to /dev/shm and the syscall memfd_create should be blocked as well.\nGeneric advice For most common services this option can be implemented and will increase the security of a service. That is, if used together with InaccessiblePaths and SystemCallFilter.\n","permalink":"https://linux-audit.com/systemd/settings/units/memorydenywriteexecute/","tags":["configuration","linux","memory","sandboxing","service hardening","systemd"],"title":"MemoryDenyWriteExecute setting"},{"categories":null,"contents":"Why and when to use InaccessiblePaths Systemd has the unit setting InaccessiblePaths to define paths that should never be accessible. Instead of using the principles of an allow list, it is an explicit deny list. It can be used to block access by a process to a location with sensitive data or a path commonly misused for exploits.\nValues Define the paths that should never be allowed access.\n[Service] InaccessiblePaths=/dev/shm When a path is prefixed with a minus (-), it is ignored if it does not exist When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.g. configured with RootDirectory) Generic advice This setting is not as powerful as others that can make larger parts of the system inaccessible, while defining just a few paths that still should be. In may still be useful when there is a need to block a very sensitive path. A good example for this is when using the MemoryDenyWriteExecute setting.\n","permalink":"https://linux-audit.com/systemd/settings/units/inaccessiblepaths/","tags":["configuration","linux","sandboxing","service hardening","systemd"],"title":"InaccessiblePaths setting"},{"categories":["Data processing"],"contents":"To remove any trailing whitespace from a file, we can use sed. By using in-place editing -i, sed can be provided with a search-and-replace action to filter out whitespace at the end of each line. By replacing it with nothing, it will effectively be removed.\nsed -i 's/[[:space:]]*$//' mytextfile.txt\nExplanation -i = inline file edit s/ = search [[:space:]]*$ = search one or more occurrences of whitespace just before the end of the line // = No text, so any occurrences of the whitespace will be emptied The [[:space:]] is called a character class and refers to space characters. Normally this includes a tab, vertical tab, form feed, new line, carriage return, and of course a space.\nNote: the usage of [[:space:]] may not work on non-Linux systems\n","permalink":"https://linux-audit.com/data-processing/faq/how-to-remove-trailing-whitespace-from-file/","tags":["data","data processing","faq","howto","linux"],"title":"How to remove trailing whitespace from a file"},{"categories":["Data processing"],"contents":"To insert a line at the beginning of a file, we can use sed to achieve this task. By using in-place editing -i, we can instruct sed to make a change to an existing file. The next step is to tell sed what to change or insert and at what place.\nsed -i '1i # New first line' mytextfile.txt\nExplanation -i = inline file edit 1i = insert at first line # New first line = Text to add ","permalink":"https://linux-audit.com/data-processing/faq/how-to-insert-line-at-the-beginning-of-file/","tags":["data processing","faq","howto","linux"],"title":"How to insert a line at the beginning of a file"},{"categories":["System Administration"],"contents":"The systemctl command has multiple options to show the memory usage. With the status subcommand followed by the service, it will show the basics, including memory usage.\nTo retrieve the information that easier to parse, then use show followed by --property=MemoryCurrent and the service name.\nUsage The status output will include memory usage.\nsystemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/nginx.service.d └─override.conf Active: active (running) since Mon 2024-06-17 17:59:45 UTC; 3h 54min ago Docs: man:nginx(8) Main PID: 36971 (nginx) Tasks: 2 (limit: 1012) Memory: 2.6M CPU: 26ms CGroup: /system.slice/nginx.service ├─36971 \u0026#34;nginx: master process /usr/sbin/nginx -g daemon on; master_process on;\u0026#34; └─36972 \u0026#34;nginx: worker process\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; \u0026#34;\u0026#34; For pulling in the information that can be parsed or scripted, consider retrieving the actual property from a running service.\n# systemctl show --property=MemoryCurrent nginx.service | awk -F= \u0026#39;{print $2}\u0026#39; 2752512 ","permalink":"https://linux-audit.com/systemd/faq/how-to-see-memory-usage-of-a-service-with-systemctl/","tags":["faq","howto","linux","memory","systemctl","systemd"],"title":"How to see memory usage of a service with systemctl?"},{"categories":["System Administration"],"contents":"The systemctl command can show settings of a systemd unit, such as a service. It can also assist in overriding these settings by using the edit subcommand followed by the unit name. This will open the editor that is configured on the system and create the override file.\nUsage Run the edit command with the unit, and the editor like vim or nano will show up.\n### Editing /etc/systemd/system/nginx.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file [Service] ProtectSystem=strict ReadWritePaths=/run /var/log/nginx ### Lines below this comment will be discarded \u0026lt;snip\u0026gt; Important: Do not remove the comments and only insert or change between the specified comment lines.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-override-a-systemd-unit/","tags":["faq","howto","linux","systemctl","systemd"],"title":"How to override the settings of a systemd unit"},{"categories":["System Administration"],"contents":"The systemctl command can be used to show all settings of an unit, such as a service. To display the full list of applicable settings, use the show subcommand followed by the unit name.\nBesides the settings, the output will also include actual runtime information, such as memory usage, when the unit was started, etc.\nUsage Just provide the unit file to see all available information.\n# systemctl show nginx.service Type=forking Restart=no PIDFile=/run/nginx.pid NotifyAccess=none RestartUSec=100ms TimeoutStartUSec=1min 30s TimeoutStopUSec=5s TimeoutAbortUSec=5s TimeoutStartFailureMode=terminate TimeoutStopFailureMode=terminate \u0026lt;snip\u0026gt; ","permalink":"https://linux-audit.com/systemd/faq/how-to-see-the-active-settings-of-a-systemd-unit/","tags":["faq","howto","linux","systemctl","systemd"],"title":"How to see the active settings of a systemd unit"},{"categories":null,"contents":"Why and when to use ReadWritePaths The setting ReadWritePaths grants read and write permissions to defined paths. It can be used in combination with other settings like \u0026lsquo;ProtectSystem=strict\u0026rsquo; to make the full file system read-only, and then open up a few paths that are required for a service to run correctly.\nValues Define the paths that are granted write access.\n[Service] ProtectSystem=strict ReadWritePaths=/run /var/log/nginx When a path is prefixed with a minus (-), it is ignored if it does not exist When a path is prefixed with a plus (+), the path is considered relative to root of directory (e.g. configured with RootDirectory) Caveats This setting will not have effect if a process is missing the normal file permissions or ownership. For additional sandboxing, consider using \u0026lsquo;CapabilityBoundingSet=~CAP_SYS_ADMIN\u0026rsquo; or \u0026lsquo;SystemCallFilter=~@mount\u0026rsquo;.\nGeneric advice When possible, restrict file system access as much as possible by implementing ProtectSystem.\n","permalink":"https://linux-audit.com/systemd/settings/units/readwritepaths/","tags":["configuration","linux","sandboxing","service hardening","systemd"],"title":"ReadWritePaths setting"},{"categories":["Nginx","System Administration","Web"],"contents":"Introduction Nginx is still a popular web server and powering a part of the web. Wouldn\u0026rsquo;t it be great if we could secure it a little bit more? In this article we use the security features to secure systemd units and services and apply it to nginx.\nIf you are not familiar yet with the unit settings of systemd, then this document would be a good introduction into the subject. Another useful resource is the nginx hardening guide, which has a focus on the nginx configuration itself.\nFile paths Like most services, nginx uses files to read and write. Typically web site content is read from the disk, while log files are created and written to. We can use the systemd unit settings to restrict access to only those paths that are strictly required for nginx to run. But how to find out what those paths are?\nConfiguration files The first step is to look at the paths configured in the nginx configuration. Usually the /etc/nginx/nginx.conf is a good start.\nNext step is following any configuration includes. If they are located in or as a subdirectory of /etc/nginx, then we can write down this path as well. Configuration files are typically read-only by the nginx process itself. Only the system administrator makes changes to it, but not from within the nginx process.\nLog files Log files are fairly easy with nginx, as they are mentioned in the configuration files.\nAnother common path is /var/log/nginx. Log files require write permissions, otherwise we can\u0026rsquo;t append new log entries to it.\nContent A web server without content is just a boring service. The locations of the content is typically defined within the configuration files. In most cases we need read-only access to these locations as well.\nOther types or locations Now we might already have discovered 90% of our locations, but maybe there are more?\nWe can use the strings command to discover words in a file, including a compiled binary. That is very useful, as on a Linux systems paths typically start with a slash followed by three characters (e.g. /var).\n# strings /usr/sbin/nginx | grep -E \u0026#34;^/[a-z]{3}\u0026#34; /lib64/ld-linux-x86-64.so.2 /var/log/nginx/error.log /etc/nginx/ /usr/share/nginx/ /etc/nginx/nginx.conf /run/nginx.pid /var/lock/nginx.lock /dev/null /var/lib/nginx/body /var/log/nginx/access.log /temp /index.html /var/lib/nginx/proxy /var/lib/nginx/scgi /var/lib/nginx/uwsgi /var/lib/nginx/fastcgi In this output we can see the default log files, a library (in /lib64), a configuration path /etc/nginx, a path for the modules and content (/usr/share/nginx), and a device (/dev/null). With this information we now have a better.\nStep by step approach Before we start adding our paths, let\u0026rsquo;s practice first what would happen if we tighten down our security profile a little bit too much. This way we know what to expect and know the actions that we can take to troubleshoot if our service is no longer working.\nKnown good state First we want to make sure that everything is working as expected, by restarting nginx. This way we have a \u0026ldquo;known good\u0026rdquo; state and know what to expect after restarting the nginx service in the future.\nsystemctl restart nginx.service\nDoes it run?\n# pidof nginx 36836 36835 We see two processes run, so that is good. Now we look in the journal to see what \u0026ldquo;good\u0026rdquo; output looks like.\n# journalctl -u nginx.service Jun 17 17:40:23 test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 17 17:40:23 test systemd[1]: Started A high performance web server and a reverse proxy server. Time to make changes and on purpose reach a failed state!\nFailing on purpose We start the configured system editor by using systemctl with the edit subcommand.\nsystemctl edit nginx.service\nImportant: when making changes, don\u0026rsquo;t remove the comment lines, only apply changes between the comment lines. This way an override file is created, then is then combined with the vendor-supplied configuration file.\nAdd ProtectSystem setting with the \u0026lsquo;strict\u0026rsquo; value.\n[Service] ProtectSystem=strict Save the file, restart nginx.\n# systemctl restart nginx.service Job for nginx.service failed because the control process exited with error code. See \u0026#34;systemctl status nginx.service\u0026#34; and \u0026#34;journalctl -xeu nginx.service\u0026#34; for details. That is not surprising, so let\u0026rsquo;s have a look what failed.\nJun 17 17:47:27 test nginx[36880]: nginx: [alert] could not open error log file: open() \u0026#34;/var/log/nginx/error.log\u0026#34; failed (30: Read-only file system) Jun 17 17:47:27 test nginx[36880]: 2024/06/17 17:47:27 [emerg] 36880#36880: open() \u0026#34;/run/nginx.pid\u0026#34; failed (30: Read-only file system) Jun 17 17:47:27 test nginx[36880]: nginx: configuration file /etc/nginx/nginx.conf test failed Jun 17 17:47:27 test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 17 17:47:27 test systemd[1]: nginx.service: Failed with result \u0026#39;exit-code\u0026#39;. Jun 17 17:47:27 test systemd[1]: Failed to start A high performance web server and a reverse proxy server. The first three lines in this output give a few good hints. The first one is that a log file could not be created or written to. The second one is that it could not create a file to store the process ID. Let\u0026rsquo;s give nginx write access to those paths using the ReadWritePaths setting.\n[Service] ProtectSystem=strict ReadWritePaths=/run /var/log/nginx Restart the nginx service and if all is good, no error should be shown. If that is the case, our first pieces of hardening is done!\nHardening measures Basics Nginx uses a master process with one or more workers. These workers are child processes and need to be spawned using the system call fork(2).\nTo avoid making the profile way too complicated or non-functioning, the generic system call set @system-service is used. This way nginx can use common systems functions, such as writing to files, forking itself, define process priority, and receiving process signals.\nRelated system calls for basic functionality bind() - Assigns address to a socket that was created with socket() dup2() - Same as dup(), duplicate file descriptor; difference is that it uses file descriptor number specified in newfd fork() - Create a new child process by duplicating the calling process, with caller becoming the parent process sendfile() - Copies data between one file descriptor and another socketpair() - Create a pair of connected sockets, for example for communication between parent and child process Capabilities and system calls used by nginx A process providing HTTP or HTTPS typically binds to port 80 and/or 443. To make this possible, the related system calls like bind(2) are need. On top of that, the capability CAP_NET_BIND_SERVICE is needed. Let\u0026rsquo;s have a look what else is needed to provide basic functionality.\nDefining securebits Within the source code of nginx we find the following piece of code with the option PR_SET_KEEPCAPS.\nif (prctl(PR_SET_KEEPCAPS, 1, 0, 0, 0) == -1) { ngx_log_error(NGX_LOG_EMERG, cycle-\u0026gt;log, ngx_errno, \u0026#34;prctl(PR_SET_KEEPCAPS, 1) failed\u0026#34;); /* fatal */ exit(2); This code section which means nginx wants to keep its capabilities. If that fails, it will quit. So this may need the use of keep-caps in SecureBits. This usage of nginx of the prctl(PR_SET_KEEPCAPS, ...) is an older style format of defining special settings with Linux capabilities.\nAlso because of the usage of prctl(2) and the usage of capabilities, we grant nginx the related capability CAP_SETPCAP.\nCPU resources sched_setaffinity(), covered by @resources (part of @system-service) setpriority(), covered by @resources (part of @system-service) No entries of realtime scheduling policies were found in the source (e.g. SCHED_DEADLINE, SCHED_FIFO, SCHED_RR). So RestrictRealtime is set to yes.\nUser and creating log files When the parent process spawns the workers, it runs them as a non-privileged user (e.g. www-data). Due to this change, the parent process needs to be able to change ownership of the user-ID and group-ID. For this reason, it requires the capabilities CAP_SETGID and CAP_SETUID.\nAnother important area is the ownership of the access and error files. To be able to create them and adjust ownership, the capabilities CAP_CHOWN and CAP_DAC_OVERRIDE are typically needed.\nNote: while testing, nginx create the access/error log with the root user as owner. When running nginx -s reopen it properly corrected them to the non-privileged user. Looks like a bug, so this part may require additional testing.\nDevice files The source code of nginx makes a reference to different files in /dev, like /dev/null, /dev/poll, and /dev/zero. When looking at an active system, only /dev/null is opened.\nUse the lsof command to validate what devices are used by your nginx processes.\nlsof -a -c nginx /dev\nWith that, we could throw multiple options into the mix.\n[Service] PrivateDevices=yes DevicePolicy=strict DeviceAllow=/dev/null Unfortunately, this will not restrict access to /dev/null alone. The PrivateDevices setting will change DevicePolicy to closed. To simplify matters, we therefore set PrivateDevices to strict.\nNamespaces Linux namespaces create an abstraction layer around processes. To determine if this is used, look at the source code for system calls like clone(2), ioctl, setns, unshare.\nSyscall clone() is mentioned in the source code, but not used. Also, it requires the a flag that starts with \u0026lsquo;CLONE_NEW\u0026rsquo;, like CLONE_NEWCGROUP for a new cgroup. With that, we can be fairly certain nginx does not create namespaces.\nTo be continued\u0026hellip; The remaining parts on hardening will be added later.\nIn the meantime, we created a predefined hardening profile for nginx.\n","permalink":"https://linux-audit.com/web/nginx-hardening-with-systemd/","tags":["cgroups","howto","linux","log files","lsof","nginx","syscall","systemd","web"],"title":"Hardening nginx with systemd security features"},{"categories":["System Administration"],"contents":"Introduction Securing a service means that one has to know what the underlying system functions and resources it uses. If we turn that around, it is as useful to know what it does not need. This information alone can be used to set boundaries of a service.\nSystemd provides a huge set of features that may be used to set boundaries. Some define functions that can be used with the help of an allow list. Not on the list? Then the related action will be denied. Other functions limit what a process can see, such information about other processes, other users, or even a limited view of the file system.\nWhy securing our services? To enable most of these features, we actually need the right set of permissions. Often it requires root permissions to lower our capabilities. Or in other words, we tell the process to give away some power.\nYou may wonder why we should invest the time in lowering the capabilities of a process. The most important reason is to stay out of trouble later on. For example, a system configuration that today is secure and fully patched, might be a weak spot in the matter of a day.\nBy configuring our processes to run with the bare minimum of permissions, we may be protected against a software vulnerability. After all, if a particular function is not used under normal conditions, it can also not be misused if we restrict access to it.\nOrganization of a systemd unit To better understand what there is to secure, let\u0026rsquo;s have a look at common service running on the web or network: nginx.\nsystemctl cat nginx.service\nThe output may look like this:\n[Unit] Description=A high performance web server and a reverse proxy server Documentation=man:nginx(8) After=network-online.target remote-fs.target nss-lookup.target Wants=network-online.target [Service] Type=forking PIDFile=/run/nginx.pid ExecStartPre=/usr/sbin/nginx -t -q -g \u0026#39;daemon on; master_process on;\u0026#39; ExecStart=/usr/sbin/nginx -g \u0026#39;daemon on; master_process on;\u0026#39; ExecReload=/usr/sbin/nginx -g \u0026#39;daemon on; master_process on;\u0026#39; -s reload ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid TimeoutStopSec=5 KillMode=mixed [Install] WantedBy=multi-user.target This unit has three sections: Unit, Service, and Install.\n[Unit] First part is the [Unit], which defines the basics of the unit, including a description. It tells the unit if it has any dependencies. As there is no point running a web server if the network is not available yet, it is defining the network-online.target as its dependencies. It also tells the service manager that in what order to start up, in this case after the network is online, file systems are available, and host and name resolution services are up and running.\n[Service] The most interesting section is the second one. Within the [Service] definition we define what we are going to do. Not just what we start, but also what we expect. Things like where we should store the process ID, if we have preparation steps before starting our main program, and even what to do in the event things go wrong. This section is also the place where we can do security tuning of our unit.\n[Install] The latest section, [Install], defines at what stage this service should be started. In this case the common multi-user target, similar to what before was runlevel 3.\nNo restrictions by default If we look the example above, we there are actually no restrictions defined. In other words, the process can do almost all it wants, unless it is restricted by file permissions, a security framework like AppArmor or SELinux, or by a generic kernel feature.\nWhy aren\u0026rsquo;t systemd units secured by default? Every system has its own needs and its own resources. The developer of the software, nginx in this case, does not know how you will be using the software. Even though the developer typically knows what system functions are used (or not), it may have limited resources to define a good basic set of required permissions. For a small script it may be easy, but a network service with many available modules, that is another story.\nNext in line is the package maintainer, the one who creates package and includes the basic unit file. This maintainer has typically less knowledge about the inner workings of the software, and struggles with the same question: what will the user exactly be doing with the software and what access or resources does it need? To make sure that the software works, he or she will be defining a basic unit file. That leaves you, as the system administrator, to do the security hardening of the service.\nAvailable security features So knowing that most systemd units have no security hardening by default, it is time to have a look at the available features that systemd has to offer. Since there is not a \u0026ldquo;one-fits-all\u0026rdquo; approach, we will be looking first at the available features. This is also the good and the bad part, there are many options available. Covering them all in one article would make it into a small book. Instead, have a look at the unit settings. That is (becoming) a long list.\nLet\u0026rsquo;s take a step back, and cover a few important things related to these security features before we start making changes.\nNot all features may be available to your system Some of the features did show up at a later systemd version that your system is running. So it\u0026rsquo;s good to check first what version you have on your system.\nsystemctl --version\nThis output will also show another important part: systemd may not be compiled with all available features. If it has support, it is prepended with a plus sign, otherwise a minus.\nAnother important part is the Linux kernel itself. While some features may be present in systemd, under the hood it usually makes use of what the Linux kernel has to offer. So if a particular mount option is not supported by the kernel, then systemd can\u0026rsquo;t make use of it. Sometimes there are workaround, but often it means that implementing this feature has to be skipped.\nImplementing requires some knowledge Another limitation of implementing the available features is the amount of knowledge about a process, how the Linux kernel works, and what this means in relation to the systemd features. Where possible, we cover this here in other articles.\nGood to know: There is a list of abbreviations in case you are not familiar with a specific term. In addition, there is a list of definitions to explain what some things refer to. If a subject requires more explanation that just a single line, often there is an article. Use the glossary or search functionality.\nChanging systemd units The location of a unit can be easily displayed by using systemctl with the subcommand cat. The first line will show this.\nsystemctl cat nginx.service\nMaking changes to these files is not recommended. These vendor-supplied files are meant to be clean and may be overwritten. Instead, we use the \u0026lsquo;drop-in\u0026rsquo; functionality, by using the edit subcommand. This will create a separate configuration file (override), and they are merged by systemd.\nsystemctl edit nginx.service\nUpon running this command, your defined system editor will open. Read the comments carefully, as only changes between the comments at the top and in the middle will be applied. To add a specific feature to our Service section, it may look like this:\n### Editing /etc/systemd/system/nginx.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file [Service] ProtectSystem=strict ### Lines below this comment will be discarded ### /lib/systemd/system/nginx.service After saving the file, you may validate the file using systemctl cat before. The changes should be reflected and are listed below the original configuration.\nApplying systemd security features to actual services With this foundation, we can start tuning actual services. An example on how to perform this is shared in harden nginx with systemd.\nPredefined hardening profiles for common services running on Linux to further secure them Software Description Service Version Apache Web server apache2 0.2 Dovecot IMAP and POP3 server dovecot 0.2alpha nginx Web server nginx 0.5 OpenSMTPD SMTP daemon originally created by developers for OpenBSD and ported to Linux and others opensmtpd 0.1 ","permalink":"https://linux-audit.com/systemd/systemd-features-to-secure-units-and-services/","tags":["linux","security","systemd"],"title":"Systemd features to secure units and services"},{"categories":null,"contents":"Why and when to use ProcSubset The systemd unit setting ProcSubset controls the \u0026ldquo;subset\u0026rdquo; mount option of /proc for the unit. By using the option, top-level entries are hidden for the process and its children.\nThe \u0026lsquo;subset=pid\u0026rsquo; was introduced in Linux 5.8.\nCaveats This function does not if the \u0026ldquo;subnet\u0026rdquo; option for procfs is not supported.\nGeneric advice The Linux kernel shares information from various kernel APIs via /proc. When activating this setting, these kernel APIs are also made unavailable, which might break common software, unless it is a trivial process. So this option is to be used with care. Typically it may be better to implement the ProtectProc setting.\n","permalink":"https://linux-audit.com/systemd/settings/units/procsubset/","tags":["configuration","linux","proc","procfs","sandboxing","systemd"],"title":"ProcSubset setting"},{"categories":null,"contents":"Why and when to use ProtectProc The setting ProtectProc aims to protect information that normally can be retrieved from /proc.\nSettings The value default, which is also the default, will not restrict access. Value invisible will hide information, where ptraceable restrict the set to only processes that be monitored with the system call ptrace(2). The value noaccess is the most strict option.\nCaveats This setting will not have effect if the kernel does not support the hidepid mount option per individual mount point.\nGeneric advice For most services use ProtectProc=invisible, as this hides information about other processes of other users. If no information about other processes from /proc is needed, then ProtectProc=noaccess can be considered.\n","permalink":"https://linux-audit.com/systemd/settings/units/protectproc/","tags":["configuration","linux","proc","procfs","sandboxing","service hardening","systemd"],"title":"ProtectProc setting"},{"categories":null,"contents":"Why and when to use RestrictAddressFamilies The setting RestrictAddressFamilies aims to restrict what socket address families can be used. When using it, the default is that it is used as an allow-list and define what address families can be used.\nSettings When this setting is not configured, there are no restrictions to what address families can be used.\nSetting the value to none will block all address families.\nTo block specific address families only, a ~ can be used to turn the allow-list into a deny-list.\nCaveats This setting does not have effect on hardware platforms like:\n32-bit x86 s390 / s390x mips / mips-le ppc / ppc-le / ppc64/ ppc64-le This setting does also not have effect on sockets created using alternative methods, including the systemd socket unit type or those created with the syscall socketpair(2).\nGeneric advice Services that use networking functionality typically use AF_INET (IPv4) and AF_INET6 (IPv6). Using the address family AF_UNIX is suggested, as it may be used for local communication between services, including the usage of syslog syscall.\nIt is advised to use SystemCallArchitectures=native in combination with this setting, to prevent easy circumvention of the restrictions.\nExample A common combination might look like this.\n[Service] SystemCallArchitectures=native RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX ","permalink":"https://linux-audit.com/systemd/settings/units/restrictaddressfamilies/","tags":["configuration","linux","networking","sandboxing","service hardening","systemd"],"title":"RestrictAddressFamilies setting"},{"categories":null,"contents":"Why and when to use ProtectHome Systemd provides the unit setting ProtectHome to protect home directories. The three paths that are included by this option are:\n/home /root /run/user Settings The default no will not restrict access to the home directories. Using yes will active full protection, not allowing access.\nThe value read-only will make the paths read-only, so no data can be written to it.\nWith tmpfs a temporary file system is being used, also read-only, yet it hides the actual home directories. It will still allow access to the actual directories when using BindPaths or BindReadOnlyPaths. For rare situations this might be useful to protect the home directories, while still allowing some very specific access.\nGeneric advice For longer running services that do not need to access home directories, use ProtectHome=yes.\n","permalink":"https://linux-audit.com/systemd/settings/units/protecthome/","tags":["configuration","linux","sandboxing","systemd","system hardening"],"title":"ProtectHome setting"},{"categories":null,"contents":"Background The Linux kernel exposes its kernel log ring buffer to userspace via /dev/kmsg and /proc/kmsg .\nWhen this setting is defined as yes, the capability CAP_SYS_MODULE will be removed from the capability bounding set. This means that all processes in the unit will no longer have access to the kernel log ring buffer.\nGeneric advice For most common services access to the kernel log ring buffer is not need, therefore safe to disable (ProtectKernelLogs=yes).\n","permalink":"https://linux-audit.com/systemd/settings/units/protectkernellogs/","tags":["capabilities","configuration","linux","log files","logging","sandboxing","systemd"],"title":"ProtectKernelLogs setting"},{"categories":null,"contents":"Explanation Kernel modules can provide additional functionality when using a modular Linux kernel, which is applicable to most systems. When this setting is set to yes, it tries to prevent the unit from loading kernel modules. This is achieved by removing the CAP_SYS_MODULE from the capability bounding set.\nGeneric advice Most units do not need the permission to load kernel modules, so typically a unit can be configured with ProtectKernelModules=yes.\n","permalink":"https://linux-audit.com/systemd/settings/units/protectkernelmodules/","tags":["capabilities","configuration","kernel modules","linux","sandboxing","service hardening","systemd"],"title":"ProtectKernelModules setting"},{"categories":["System Administration"],"contents":"The control group of a process can be retrieved from the /proc directory. We only need to know the PID of the process, which can be found using ps or pidof.\nUsage If we know that our PID is 1234, then showing the cgroup is as easy as using cat to see the contents of the \u0026lsquo;cgroup\u0026rsquo; file.\ncat /proc/1234/cgroup\nTo see the cgroup for the nginx process (or one of them), we could something like this.\ncat /proc/$(pidof -s nginx)/cgroup\nWant to see the value directly from the process listing?\nRelevant FAQ: How to see cgroup in ps output?\n","permalink":"https://linux-audit.com/processes/faq/how-to-see-cgroup-of-a-process/","tags":["cgroups","faq","howto","linux","processes"],"title":"How to see the cgroup of a process"},{"categories":["System Administration"],"contents":"The ps command can show the control group of a process using the -o option, followed by the right column names.\nUsage To show processes and the control group, we can filter the output columns.\n# ps -e -o pid,cgroup:64,args PID CGROUP COMMAND 1 0::/init.scope /lib/systemd/systemd --system --deserialize 58 2 - [kthreadd] 3 - [rcu_gp] \u0026lt;snip\u0026gt; 576 - [xprtiod] 634 0::/system.slice/dbus.service @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 640 0::/system.slice/networkd-dispatcher.service /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers 645 - [nfsiod] 653 0::/system.slice/systemd-logind.service /lib/systemd/systemd-logind 696 0::/system.slice/system-getty.slice/getty@tty1.service /sbin/agetty -o -p -- \\u --noclear tty1 linux In this example the PID is displayed, with the control group, and the command with its arguments. By specifying the width of 64 characters, we can properly see the full name of the control group.\n","permalink":"https://linux-audit.com/processes/faq/how-to-see-cgroup-in-ps-output/","tags":["cgroups","faq","howto","linux","processes"],"title":"How to see cgroup in ps output"},{"categories":["System Administration"],"contents":"The timedatectl command can show the time, time zone information, and its status. Add the timesync-status subcommand to see synchronization details.\nUsage Use timedatectl with the timesync-status command to see the actual status. Under normal conditions, the leap should show \u0026rsquo;normal'.\n# timedatectl timesync-status Server: 185.125.190.56 (ntp.ubuntu.com) Poll interval: 34min 8s (min: 32s; max 34min 8s) Leap: normal Version: 4 Stratum: 2 Reference: 4FF33C32 Precision: 1us (-25) Root distance: 762us (max: 5s) Offset: +882us Delay: 15.169ms Jitter: 7.098ms Packet count: 711 Frequency: +2.731ppm ","permalink":"https://linux-audit.com/systemd/faq/how-to-see-time-synchronization-details-with-timedatectl/","tags":["faq","howto","linux","systemd","time","time synchronization"],"title":"How to see the time synchronization details with timedatectl"},{"categories":["System Administration"],"contents":"With the hostnamectl command basic system information like the operating system, hostname, and machine ID can be displayed.\nUsage Run the command without any parameters to get the status displayed, including the machine ID.\nhostnamectl\n","permalink":"https://linux-audit.com/systemd/faq/how-to-show-systemd-machine-id/","tags":["faq","hostnamectl","linux","systemd"],"title":"How to show the systemd machine ID"},{"categories":["System Administration"],"contents":"The systemctl command can be used to show dependencies between units with the list-dependencies subcommand. A nicely human-readable output will be displayed showing the selected unit, followed by the dependencies that rely on this unit. This is useful when a unit is in a failed state due to a dependency on another unit.\nUsage To see which units require the multi-user target to be active:\n# systemctl list-dependencies multi-user.target multi-user.target ● ├─apport.service ● ├─console-setup.service ● ├─cron.service ● ├─dbus.service ○ ├─dmesg.service ○ ├─e2scrub_reap.service ○ ├─grub-common.service ○ ├─grub-initrd-fallback.service ○ ├─irqbalance.service \u0026lt;snip\u0026gt; Want to move up the other way and see on which our unit requires? Add the --reverse option.\n# systemctl list-dependencies --reverse multi-user.target multi-user.target ● └─graphical.target For automated processing or just plaintext output, use the --property option.\n# systemctl show --no-pager --value --property=\u0026#34;Wants\u0026#34; multi-user.target lxd-agent.service ubuntu-advantage.service snapd.recovery-chooser-trigger.service e2scrub_reap.service snap-core20-2264.mount snap.lxd.activate.service apport.service \u0026lt;snip\u0026gt; Interesting keywords to use as the property are:\nAfter Before ConflictedBy Conflicts RequiredBy Requires WantedBy Wants ","permalink":"https://linux-audit.com/systemd/faq/how-to-see-the-dependencies-of-a-systemd-unit/","tags":["faq","linux","systemctl","systemd"],"title":"How to see the dependencies of a systemd unit"},{"categories":["System Administration"],"contents":"The systemctl command can show the available systemd unit types when using the option --type=help.\nUsage # systemctl --type=help Available unit types: service mount swap socket target device automount timer path slice scope To use a particular type, define one of the available unit types as the value, such as a service.\nsystemctl list-units --type=service\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-available-systemd-unit-types/","tags":["faq","linux","systemctl","systemd"],"title":"How to see the available systemd unit types"},{"categories":["System Administration"],"contents":"The systemctl command will show by default all active units. To filter down on a particular unit type, use the --type= option, followed by the type. Not sure what types are available? Run systemctl --type=help.\nUsage systemctl list-units --type=target\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-all-active-units-of-one-type/","tags":["faq","linux","systemctl","systemd"],"title":"How to see all active systemd units of one type"},{"categories":["System Administration"],"contents":"To limit the maximum size that journals may use on the system, define the setting SystemMaxUse in /etc/systemd/journald.conf . Save the file, confirm that the settings are correct, then restart the journal daemon.\nConfiguration Open /etc/systemd/journald.conf , copy the commented line, remove the hash, and assign it a value.\nSystemMaxUse=256M Note: depending on how many events happen on a system, this value might be too small. Make sure that the size for logs is big enough.\nCheck configuration and restart daemon Confirm that the settings are correct using the systemd-analyze command with the subcommand cat-config.\nsystemd-analyze cat-config systemd/journald.conf\nIf all looks good, restart the journald daemon.\nsystemctl restart systemd-journald\nFinally confirm the disk usage. If you defined a lower value than the disk usage before, then it should have an updated size.\njournalctl --disk-usage\nThe new maximum size should also be reflected in the output within the journal itself. For example, if we lower it to 256 megabyte, the maximum should displayed as such.\n# journalctl -u systemd-journald.service --since=\u0026#34;today\u0026#34; | grep -E \u0026#34;(Runtime|System) Journal\u0026#34;` Jun 15 21:18:36 web.example.org systemd-journald[75881]: System Journal (/var/log/journal/1c934add5be04ca8e30d000071964675) is 95.4M, max 4.0G, 3.9G free. Jun 15 22:39:27 web.example.org systemd-journald[75891]: System Journal (/var/log/journal/1c934add5be04ca8e30d000071964675) is 95.4M, max 256.0M, 160.5M free. ","permalink":"https://linux-audit.com/systemd/faq/how-to-limit-the-disk-usage-of-the-systemd-journal/","tags":["faq","journalctl","linux","logging","systemd","systemd-analyze"],"title":"How to limit the disk usage of the systemd journal"},{"categories":["System Administration"],"contents":"The journalctl command can be used to show the journal. By using the --disk-usage option, the size of the journal is displayed. This includes the archived and active journal files. When the journal is using too much disk space, consider performing a vacuum task.\nUsage Showing the disk usage is quick and easy.\n# journalctl --disk-usage Archived and active journals take up 56.0M in the file system. Does the journal take up too much space?\n# journalctl --vacuum-size=50M Deleted archived journal /var/log/journal/d8bd6473290d43a9942eaba0a506a454/system@ca889eb2eae24e41b37a50d33bad131c-0000000000000001-00060ed90326924f.journal (8.0M). Deleted archived journal /var/log/journal/d8bd6473290d43a9942eaba0a506a454/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000004e2-00060ed9041f690a.journal (8.0M). Deleted archived journal /var/log/journal/d8bd6473290d43a9942eaba0a506a454/system@ca889eb2eae24e41b37a50d33bad131c-0000000000000f7b-000615baadbd8a6a.journal (24.0M). Vacuuming done, freed 40.0M of archived journals from /var/log/journal/d8bd6473290d43a9942eaba0a506a454. Vacuuming done, freed 0B of archived journals from /run/log/journal. Vacuuming done, freed 0B of archived journals from /var/log/journal. Note: even when defining a specific value, the usage may be a little bit higher than the defined threshold. This is most likely due to overhead on the binary database structure and reserving blocks for performance.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-size-of-systemd-journal/","tags":["faq","journalctl","linux","systemd"],"title":"How to see the size of the systemd journal"},{"categories":["System Administration"],"contents":"The journalctl command can show all events related to the kernel itself using the --dmesg option. This option will filter out kernel messages and has a similar output as the dmesg command.\nUsage Use the full or shorter option to query the kernel messages.\njournalctl -k\nLooking for only the kernel messages of today? Combine it with the --since= option.\njournalctl -k -S \u0026quot;today\u0026quot;\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-kernel-messages-with-journalctl/","tags":["faq","journalctl","kernel","linux","logging","systemd"],"title":"How to see kernel messages with journalctl"},{"categories":["System Administration"],"contents":"Systemd units define resources that can be used by the system. Examples of these units are a service, path, socket, and timer. Each unit type has its own basic set of properties that then individually can be configured. Unit types can be recognized by their file extension. A service will use the \u0026lsquo;.service\u0026rsquo; extension, making it easy to recognize. Units are usually managed with the systemctl command.\nSee systemd unit types and their purpose for a full overview of the units.\n","permalink":"https://linux-audit.com/systemd/faq/what-is-a-systemd-unit/","tags":["faq","linux","systemctl","systemd"],"title":"What is a systemd unit?"},{"categories":["System Administration"],"contents":"The journalctl command shows by default the oldest entries it has in the journal. Typically we are not interested in that, for that purpose there is the --since= option. This option defines that entries should be after the specified moment in time. Besides using an actual date, a shortened name like \u0026rsquo;today\u0026rsquo; can also be used that automatically defines the date and time.\nUsage To see the entries of today, use the aptly named \u0026rsquo;today'.\njournalctl --since=\u0026quot;today\u0026quot;\nLooking for a very recent entry? Define a period in minutes.\njournalctl --since=\u0026quot;15 min ago\u0026quot;\nWant to be really precise? Define the date and time.\njournalctl --since=\u0026quot;2024-06-15 02:15:30\u0026quot;\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-only-recent-journal-entries/","tags":["faq","journalctl","linux","systemd"],"title":"How to see only recent journal entries"},{"categories":["System Administration"],"contents":"The journalctl command can show the events from its journal by --unit= followed by the service or its unit name. This way events will be filtered, making it much easier to troubleshoot a particular service.\nExample journalctl -u nginx.service\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-logging-for-a-specific-unit-or-service/","tags":["faq","journalctl","linux","systemd"],"title":"How to see logging for a specific unit or service"},{"categories":["System Administration"],"contents":"The journalctl command can show continuously new log entries with the --follow option. When new entries are added to the journal, they are automatically shown.\nUsage The follow option is a great option to continuously monitor a particular unit.\njournalctl --follow --unit=nginx.service\nWithout providing a unit, all system events will be shown and followed.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-new-log-entries-automatically-with-journalctl/","tags":["faq","journalctl","linux","systemd"],"title":"How to see new log entries automatically with journalctl"},{"categories":["System Administration"],"contents":"When changes are made to systemd unit files, such as a .service file, the systemd daemon needs to be reloaded. This way it can activate the changes. The reload is done use the systemctl command followed by daemon-reload.\nsystemctl daemon-reload\n","permalink":"https://linux-audit.com/systemd/faq/how-to-reload-the-systemd-configuration/","tags":["faq","linux","systemctl","systemd"],"title":"How to reload the systemd configuration"},{"categories":["System Administration"],"contents":"Systemd units that are in a masked state are administratively disabled. While being in this state, they can not be started until they are unmasked. Typically a unit is masked when it should not start by default or manually, to prevent it causing issues or running an unwanted service. With systemctl and the subcommand mask, a systemd unit can be masked.\nRelevant FAQ: How to see all masked units with systemctl?\n","permalink":"https://linux-audit.com/systemd/faq/what-is-a-masked-systemd-unit/","tags":["faq","linux","systemctl","systemd"],"title":"What is a masked systemd unit?"},{"categories":["System Administration"],"contents":"Systemd is a system and service manager. The name is short for \u0026lsquo;system daemon\u0026rsquo;, an ongoing service that manages the system. As it is also a service manager, it is responsible for start, stopping, and monitoring services. Systemd replaces the SysV init system and focuses on performance and resource management. It was created by Lennart Poettering in 2010, with Fedora Linux being the first to adopt it in May 2011. In 2015, several major Linux distributions started shipping with systemd.\nIf you are running a modern Linux distribution, changes are high that systemd is being used. This can be confirmed by using a command like systemctl on the command-line.\nSystemd units An important part of systemd are its unit files. These files define how components interact with each other, including the definition of a service or a mount point.\nBenefits of using systemd Faster boot Traditional init systems started services sequentially, which meant a service sometimes had to wait for another. Systemd performs a analysis of service dependencies and starts services independently. Due to this parallel start, the boot time decreases. An example of this is that while nginx should wait for the network to be up, it does not have to wait for a file system check that can be done in the background.\nGeneric service management tooling Previously a Linux system had a set of scripts to start, stop, and restart services. Depending on the service, it was very basic or more advanced. Systemd uses a more generic way of defining services in the form of unit files. Each of these file define how the service should stop and start, optionally complemented with extra functions. Due to this new approach, it can better track the state of services, keep relevant event logs, and provide additional details.\nBetter resource management Normal scripts in the traditional init systems had almost no capabilities to resource usage by services. With systemd making use of cgroups, it can monitor resource usage and set clear limitations. This increases the stability of the system and allows system administrators to earlier detect issues.\nCentralized logging Systemd offers a single place to store all log events, from the kernel, up to the services. With a binary format and storing metadata (like timestamps and process IDs), it allows for performing queries to retrieve specific data. This allows for quickly finding relevant entries for a particular service, or events from a specific moment in time.\n","permalink":"https://linux-audit.com/systemd/faq/what-is-systemd/","tags":["faq","linux","systemctl","systemd"],"title":"What is systemd?"},{"categories":["SSH","System Administration"],"contents":"The ProxyJump defines a bastion host (jump host, jump server, jump box) to use.\nValues Value Meaning none Disable ProxyJump functionality HOST Define the hostname of the bastion host [USER]HOST[:PORT] Define one or more parameters of the bastion host URI Define parameters in URI format Hostname Format: hostname\nUser Format: user@hostname\nPort Format: hostname:port\nCommand-line usage ssh -J bastion destinationsystem\nThese parameters can also be specified in the format as a URI .\nssh -J ssh://user@hostname:port destinationsystem\nConfiguration file While command-line usage is available, typically the settings for a bastion host are stored in your local SSH configuration file.\nHost jumphost HostName jumphost.example.com Host webserver HostName webserver.example.com ProxyJump jumphost ProxyCommand versus ProxyJump Before the ProxyJump option was available, jumping was done via the ProxyCommand option.\nWhen both options are set, then the first one available will be used.\n","permalink":"https://linux-audit.com/ssh/config/client/option-proxyjump/","tags":["linux","openssh","ssh","ssh client","ssh-agent"],"title":"SSH ProxyJump option"},{"categories":["SSH","System Administration"],"contents":"The ForwardAgent option specifies if SSH agent forwarding is allowed or not.\nForwardAgent values Value Meaning Yes Agent forwarding is allowed No (default) Agent forwarding is not allowed PATH Path to the agent socket $VARIABLE Environment variable that stores the path Security caution Agent forwarding should not be used if not strictly needed. Any user that can access the agent\u0026rsquo;s socket stored in SSH_AUTH_SOCK may have access through the forwarded connection. While key material may not be accessible, the keys can still be used to authenticate to any of the identities that are active in the SSH agent.\nIf you really need agent forwarding, use it one a single session basis with the -A option.\nssh -A user@host\nWhen using a bastion host (jump host, jump server, jump box), consider using ProxyJumpoptional as this is a safer alternative.\n","permalink":"https://linux-audit.com/ssh/config/client/option-forwardagent/","tags":["linux","openssh","ssh","ssh client","ssh-agent"],"title":"SSH ForwardAgent option"},{"categories":["System Administration"],"contents":"The agent forwarding feature in SSH allows using your local SSH agent to be reached through an existing SSH connection. This way you don\u0026rsquo;t have to store copies of your private keys on intermediate systems to use them for authentication. While SSH agent forward simplifies things, it also introduces a new risk related to Unix domain socket. If a user on the intermediate system can access the related socket, then it may abuse this connection back to the SSH agent to authenticate on your behalf.\nSee the ForwardAgent option for more details.\n","permalink":"https://linux-audit.com/ssh/faq/what-is-ssh-agent-forwarding/","tags":["authentication","faq","howto","linux","openssh","ssh","ssh-agent"],"title":"What is SSH agent forwarding?"},{"categories":["System Administration"],"contents":"The ssh-agent command is started manually using eval $(ssh-agent). This will initiate the SSH agent and make it available for clients, such as ssh, to use it.\nTo confirm that the agent is running is by looking at the SSH_AUTH_SOCK environment variable.\nAutomatic start of SSH agent Gnome Keyring SSH Agent When using Gnome, it typically comes with its SSH agent as part of Keyring. This will automatically load any files in ~/.ssh when both the secret and public key is available.\nSystemd service unit Create a unit file (/etc/systemd/system/ssh-agent.service):\n[Unit] Description=SSH key agent [Service] Type=simple Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK [Install] WantedBy=default.target Then enable it.\nsystemctl enable --now ssh-agent.service\nStart via login profile When using bash, add the following snippet to your profile ~/.bash_profile to active it upon login.\nif [ -z \u0026#34;$SSH_AUTH_SOCK\u0026#34; ] ; then eval $(ssh-agent -s) ssh-add fi This checks for the presence of the environment variable. If it is empty (-z), then it will start the agent and add keys.\n","permalink":"https://linux-audit.com/ssh/faq/how-to-start-ssh-agent/","tags":["authentication","faq","howto","linux","openssh","ssh","ssh-agent"],"title":"How to start the SSH agent?"},{"categories":["Passwords","System Administration"],"contents":"The ssh command can be instructed to avoid using the SSH agent and its known identities with the -o IdentityAgent=none option. This sets the option IdentityAgent to disable the SSH agent.\nCommand line usage To use this option on the command line, specify the setting and set it to no.\nssh -oStrictHostKeyChecking=no user@hostname\nUsage in configuration file This setting can also be configured in /etc/ssh_config to define it as its default. An alternative is to define it per host, in the file in your home directory ~/.ssh/config .\nTo make it a default, define a generic section at the bottom of your config file.\nHost * # Only use the Identities that are specified with IdentityFile IdentitiesOnly yes # Define primary key IdentityFile /home/%u/.ssh/id_ed25519 # Disable entities known to ssh-agent IdentityAgent none # Avoid password authentication PasswordAuthentication no ","permalink":"https://linux-audit.com/ssh/faq/how-to-disable-usage-of-ssh-agent/","tags":["authentication","ed25519","faq","howto","linux","openssh","ssh","ssh-agent"],"title":"How to disable the usage of the SSH agent"},{"categories":["System Administration"],"contents":"The ssh-agent command starts the SSH agent, a helper utility to store private keys when using public key authentication. The ssh-agent process is usually started at the the beginning of a login session and then can be connected to by a SSH client. Clients can detect the environment variable named SSH_AUTH_SOCK.\nRelated settings on the client IdentityAgent ","permalink":"https://linux-audit.com/ssh/faq/what-is-the-purpose-of-ssh-agent/","tags":["authentication","faq","howto","linux","openssh","ssh","ssh-agent"],"title":"What is the purpose of the SSH agent?"},{"categories":["SSH","System Administration"],"contents":"The IdentityAgent option specifies what UNIX-domain socket to use to communicate with the authentication agent. When configured, it overrides the environment variable SSH_AUTH_SOCK and provides the option to select a specific agent.\nBesides the option to define a socket, the location of the socket can also be provided by the SSH_AUTH_SOCK environment variable. In that case the value should defined as \u0026ldquo;SSH_AUTH_SOCK\u0026rdquo; (without quotes). If the value starts with a \u0026lsquo;$\u0026rsquo;, it indicates that another environment variable is to be used.\nTo disable the usage of the SSH-agent, define the value \u0026ldquo;none\u0026rdquo; (without quotes).\nHost * # Disable entities known to ssh-agent for all systems IdentityAgent none ","permalink":"https://linux-audit.com/ssh/config/client/option-identityagent/","tags":["linux","openssh","ssh","ssh client"],"title":"SSH IdentityAgent option"},{"categories":["System Administration"],"contents":"The file /etc/ssh/ssh_config is the main configuration file of the OpenSSH server daemon (sshd).\nPurpose It defines the system-wide settings for the SSH client. It can be overridden by storing settings in ~/.ssh/config that takes precedence.\nConfiguration To learn more about the available configuration settings, have a look at the section OpenSSH client configuration.\n","permalink":"https://linux-audit.com/system-administration/files/etc-ssh-ssh_config/","tags":["configuration","linux","ssh","ssh client","ssh_config"],"title":"/etc/ssh/ssh_config: SSH client configuration"},{"categories":["SSH","System Administration"],"contents":"For the SSH client there are typically two places where configuration files are stored: in the home directory of the user and a global configuration file.\nUser configuration Location: ~/.ssh/config This file is stored in the home directory of an user. It is optional and by default no file is available.\nSystem-wide configuration Location: /etc/ssh/ssh_config Overrides via: /etc/ssh/ssh_config.d/*.conf The default settings are in the ssh_config file. The system administrator may add customizations to this file, but typically it is advised to override settings using a separate configuration file. This way settings are not being overwritten by accident when a new package is released.\nOrder of precedence The SSH client has a defined order of precedence when it comes to evaluating settings.\nCommand-line options Configuration file of user Configuration file of system Options defined on the command-line will have the highest priority and therefore override settings in configuration files. The user configuration will override the settings in the system-wide configuration.\n","permalink":"https://linux-audit.com/ssh/config/client/configuration-files/","tags":["linux","openssh","ssh","ssh client","ssh_config"],"title":"SSH configuration files"},{"categories":["Software","System Administration"],"contents":"To see all installed packages on systems like Arch Linux, run the pacman command with the --query option.\na52dec 0.8.0-2 aalib 1.4rc5-18 abseil-cpp 20240116.2-2 accountsservice 23.13.9-2 acl 2.3.2-1 acpi 1.7-3 acpid 2.0.34-1 ..snip.. zstd 1.5.6-1 zvbi 0.2.42-1 zxing-cpp 2.2.1-1 ","permalink":"https://linux-audit.com/software/package-manager/faq/how-to-show-installed-packages-with-pacman/","tags":["faq","howto","linux","package manager","software"],"title":"How to show all installed packages with pacman"},{"categories":["Passwords","SSH","System Administration"],"contents":"One of the common methods to authenticate with a SSH server is using the combination of a username and password. With the option PasswordAuthentication we can define if we want to use this type of authentication. While yes is the default, it might be useful to disable it for hosts that require public key authentication. This way we instruct the client to only try that.\n","permalink":"https://linux-audit.com/ssh/config/client/option-passwordauthentication/","tags":["linux","openssh","ssh","ssh client"],"title":"SSH PasswordAuthentication option"},{"categories":["SSH","System Administration"],"contents":"The SSH client won\u0026rsquo;t connect to a system when it sees that host key changed since the initial connection it made. This helps against MitM attacks. The client knows when the host key is different by comparing it with the related values in the ~/.ssh/known_hosts file.\nValues Value Automatically save new host keys Action if host key changed yes No Refuse ask No, ask Refuse accept-new Yes Refuse no | off Yes Connect When connecting to many different systems, the accept-new value can help reducing the manual step to accept keys.\nDefault value By default this option is set to ask. This is a sane default, that is suitable for most systems.\nWhen to use Disabling this check is normally not advised, as there is typically a good reason why host keys change. Maybe the system administrator replaced the host keys by another type, but it might also be a deliberate attack. So use this option only in trusted environments where the risks are low.\n","permalink":"https://linux-audit.com/ssh/config/client/option-stricthostkeychecking/","tags":["linux","openssh","ssh","ssh client"],"title":"SSH StrictHostKeyChecking option"},{"categories":null,"contents":"What is security through obscurity? Security through obscurity (STO) is hiding, masking, or concealing parts of a system to enhance its security. By itself this does not increase the security level, but it can be an effective method in combination with a layered security defense. So it complements the overall security efforts, but alone is it not to be considered a real security measure.\nAnother part of security through obscurity is the principle of hiding in plain sight. Like one can use a password and hide that in a creative way into a cooking recipe.\nLinux examples of security through obscurity Let\u0026rsquo;s have a look at some examples on Linux.\nReplace application name If you are running nginx, you could decide to use Apache in a HTTP header. This may help automated scanners that are looking for nginx to ignore your system. Another option is that an attacker might be confused.\nChange port number A good example of changing the port number is that of changing the SSH server to another port. It may reduce automated scans, brute-force authentication attempts, and decrease the number of log entries.\nAdding extra (fake) ports to the system Most secured systems will run a minimum of services. That also means that the number of open ports is often limited to just a few. To hide the services in plain sight, one could add additional ports to the system, running a fake service, or even a honeypot. Actions against these ports or the honeypot, may be logged and used as a warning signal. A bit like the canary in the coal mine to discover problems early.\n","permalink":"https://linux-audit.com/security-concepts/security-through-obscurity/","tags":["linux","security","security concepts"],"title":"Security Through Obscurity (STO)"},{"categories":["System Administration"],"contents":"To stop all processes of a single user, the kill command can be used. An easier alternative is using killall with the --user option, followed by the user name.\nExample:\nkillall -u michael\nThis command, with the shortened option, will send by default the SIGTERM signal to the process. By using the --signal option, another process signal can be provided.\n","permalink":"https://linux-audit.com/processes/faq/how-to-stop-all-processes-of-a-single-user/","tags":["faq","howto","linux","processes"],"title":"How to stop all processes of a single user"},{"categories":["System Administration"],"contents":"The ssh command can be instructed to ignore incorrect host keys with the -o StrictHostKeyChecking=no option. This sets the option StrictHostKeyChecking to ignore changed host keys.\nThis setting can also be configured in /etc/ssh_config to define it as its default. An alternative is to define it per host, in the file in your home directory ~/.ssh/config .\nCommand line usage To use this option on the command line, specify the setting and set it to no.\nssh -o StrictHostKeyChecking=no user@hostname\nPossible security impact Disabling this check may increase the risk of a MitM attack. When possible, try to confirm why the host key changed. When using a hostname, confirm first that the name resolution is correct.\n","permalink":"https://linux-audit.com/ssh/faq/how-to-ignore-host-authenticity-and-key-fingerprint/","tags":["authentication","faq","hostname","howto","linux","openssh","ssh"],"title":"How to disable the SSH host key check?"},{"categories":["SSH","System Administration"],"contents":"Why change your SSH port? Systems that are available via the internet and can\u0026rsquo;t be fully protected with a firewall, they might benefit from running on a different TCP port than the default 22. This way automated scanners will less likely probe your system(s), as they don\u0026rsquo;t know what port you use for SSH.\nChanging your SSH port won\u0026rsquo;t make a system more secure in itself, and therefore is often called security through obscurity. At the same, it may help in reducing noise in your logs, making it easier to monitor. This in itself slightly improves security, as system administrators typically start to ignore log files if they are flooded with authentication failures.\nChange SSH port number The server configuration is typically stored in /etc/ssh/sshd_config . If you have a /etc/ssh/sshd_config.d directory, then typically it a good idea to make your changes there. Settings will then override the main configuration file. Create a new file, such as 99-custom.conf.\nPort 2222 Test configuration After making changes, test if all is good. If there is an issue, then the output might look like this.\n# sshd -t /etc/ssh/sshd_config.d/99-custom.conf: line 3: Bad configuration option: Portt /etc/ssh/sshd_config.d/99-custom.conf: terminating, 1 bad configuration options Update your firewall If you are running a firewall, then this is the time to add the new port.\nFirewall Command FirewallD firewall-cmd \u0026ndash;permanent \u0026ndash;zone=public \u0026ndash;add-port=2222/tcp \u0026amp;\u0026amp; firewall-cmd \u0026ndash;reload iptables /sbin/iptables -A INPUT -m state \u0026ndash;state NEW -m tcp -p tcp \u0026ndash;dport 2222 -j ACCEPT UFW ufw allow 2222/tcp Don\u0026rsquo;t remove the existing port 22 yet, as we are currently connected to it.\nUpdate SELinux If SELinux is enabled, then update the configuration.\nsemanage port -a -t ssh_port_t -p tcp 2222\nNot sure if SELinux is enabled?\n# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 Restart SSH daemon Next step is restarting the SSH daemon. Under normal conditions you should stay connected on the active connection.\nsystemctl restart ssh.service\nConfirm that the port is active in the new configuration.\nsshd -T | grep port\nIf the configuration setting is correct, then connect to the system via another session using the newly defined port.\nssh -p 2222 192.168.1.250\nThat\u0026rsquo;s it!\n","permalink":"https://linux-audit.com/ssh/change-ssh-server-port/","tags":["linux","openssh","selinux","ssh","ssh daemon","sshd_config"],"title":"Change SSH server port number"},{"categories":["SSH","System Administration"],"contents":"Configure a Message of the Day The MOTD is typically stored in /etc/motd or a related directory, such as /etc/update-motd.d or /etc/motd.d/. The related message or messages are then displayed after a user is logged in.\nAnother option is that the MOTD is displayed using the PAM configuration. To see if this is the case, perform a grep in your PAM configuration directory.\ngrep -ir motd /etc/pam.d | grep -v \u0026quot;:#\u0026quot;\nIf PAM is not used to define the Message of the Day, then it can be enabled via the SSH configuration.\nConfiguration changes to can be done in the main SSH configuration file, but it is better to add them to a file in the /etc/ssh/sshd_config.d directory. This way it will override the changes and don\u0026rsquo;t get overwritten when a package is receiving updates.\nFor example, edit /etc/ssh/sshd_config.d/99-custom.conf and add:\nPrintMotd yes To activate, test the changes and restart the SSH daemon. Let\u0026rsquo;s first test the new configuration.\nsshd -t\nAll well? Then restart the SSH daemon.\nsystemctl restart ssh.service\nConfigure a banner The MOTD is displayed after the user logged in. The banner on the other hand, is displayed before authentication.\nNormally this type of banner is configured in /etc/issue.net for non-local logins. For example:\n===================================== Unauthorized access is prohibited ===================================== * This system is monitored * Actions are logged ===================================== After saving the file, set the new option to a custom SSH configuration file (see above).\nBanner /etc/issue.net Test the configuration.\nsshd -t\nRestart the SSH daemon after testing the configuration succeeded.\nsystemctl restart ssh.service\nThen check if the file is properly configured.\nsshd -T | grep banner\nIf the configuration setting is correct, then connect to the system via another session.\n# ssh 192.168.1.250 ===================================== Unauthorized access is prohibited ===================================== * This system is monitored * Actions are logged ===================================== michael@192.168.1.250\u0026#39;s password: That\u0026rsquo;s it!\n","permalink":"https://linux-audit.com/ssh/configure-ssh-welcome-message-or-banner/","tags":["linux","openssh","pam","ssh","ssh daemon"],"title":"Configure a SSH welcome message or banner"},{"categories":["SSH","System Administration"],"contents":"Escape sequences are a patterns of keys that are recognized to send special instructions. They can be also used to debug issues with the connection.\nTo see the ones that are supported, press tilde (~) followed by a question mark (?).\nSupported escape sequences: ~. - terminate connection (and any multiplexed sessions) ~B - send a BREAK to the remote system ~C - open a command line ~R - request rekey ~V/v - decrease/increase verbosity (LogLevel) ~^Z - suspend ssh ~# - list forwarded connections ~\u0026amp; - background ssh (when waiting for connections to terminate) ~? - this message ~~ - send the escape character by typing it twice (Note that escapes are only recognized immediately after newline.) Important: you may need to press enter first before sending the escape sequence.\nTerminate connection When using a tilde followed by a dot, the connection will be closed. This is very helpful when your connection is stuck and no longer responds to CTRL+C. Especially connections are firewalls might encounter this issue, when the underlying network connection is terminated. The client can no longer send or receive data, so it gets stuck.\nList forwarded connections Using a tilde with a hash (#), the forwarded or open connections are displayed.\nExample output:\nThe following connections are open: #0 client-session (t4 r0 i0/0 o0/0 e[write]/4 fd 4/5/6 sock -1 cc -1 io 0x01/0x01) Troubleshooting an active SSH session Want to troubleshoot an active session? Then the tilde followed by a \u0026lsquo;v\u0026rsquo; might help with more verbose output.\nWhen pressing the combination, you may see output like\n~v [LogLevel VERBOSE] Once more and it will go to DEBUG, then DEBUG2, finally DEBUG3.\n","permalink":"https://linux-audit.com/ssh/ssh-escape-sequences/","tags":["debugging","linux","openssh","ssh","troubleshooting"],"title":"SSH escape sequences"},{"categories":["System Administration"],"contents":"If a SSH connection is stuck and no longer responds to CTRL+C, try using the built-in escape sequences. First send a tilde (~) followed by a dot (.). Normally this should stop the connection.\nWhen using escape sequences, you should not see the tilde and dot. If that happens, press first enter, so the client is a state where escape sequences can be used.\n","permalink":"https://linux-audit.com/ssh/faq/how-to-terminate-ssh-connection-that-does-not-respond-to-ctrl-c/","tags":["authentication","faq","howto","linux","openssh","ssh"],"title":"How to terminate a SSH connection that does not respond to CTRL+C"},{"categories":["System Administration"],"contents":"The passphrase on a SSH key can be removed using the ssh-keygen command with the -p. This will interactively ask for the existing passphrase and provides the option to clear it.\n# ssh-keygen -p -f /path/to/key-file Enter old passphrase: Key has comment \u0026#39;mykey\u0026#39; Enter new passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved with the new passphrase. ","permalink":"https://linux-audit.com/ssh/faq/how-to-remove-passphrase-from-ssh-key/","tags":["authentication","faq","howto","linux","openssh","ssh"],"title":"How to remove the passphrase from a SSH key"},{"categories":["System Administration"],"contents":"The OpenSSH authentication agent stores the SSH identities and may use them when connecting to a SSH server. With the ssh-add command and -l option the fingerprints of the known identities can be shown.\nssh-add -l\nIf the agent is not running, then you will see the message \u0026lsquo;Could not open a connection to your authentication agent.\u0026rsquo;.\n","permalink":"https://linux-audit.com/ssh/faq/how-to-see-available-ssh-keys-in-openssh-agent/","tags":["authentication","faq","howto","linux","openssh","ssh"],"title":"How to see the available SSH keys in the OpenSSH authentication agent"},{"categories":["System administration"],"contents":"The kill command is used on Linux to send a process signals. This can be a numeric value or its defined name (e.g. SIGTERM).\n","permalink":"https://linux-audit.com/system-administration/commands/kill/","tags":["kill","linux","processes"],"title":"kill"},{"categories":["System Administration"],"contents":"A zombie process, or defunct process, has completed execution, but has still an entry in the process table. The process is considered to be terminated, but lacks the proper house keeping to reflect this state.\nNormally when a parent process spawns child processes, it will have to use the wait(2) or waitpid(2) function. This way when a child exists, the parent will know about it. If no processes waits for a child process, then you get a zombie process. It does no longer take up system resources, but at the same time it stays around. One of the options is sending a SIGCHLD signal to its parent, as this may give the kernel the green light to dispose of the zombie process.\nStopping zombie processes To clean up a zombie process, try the steps in this article:\nRelevant FAQ: How to kill a zombie process?\n","permalink":"https://linux-audit.com/processes/faq/what-is-a-zombie-process/","tags":["faq","howto","linux","processes"],"title":"What is a zombie process?"},{"categories":["System Administration"],"contents":"When you have a zombie process that does not respond to kill -9, then we can try a few more options.\nOption 1: sending a signal to the parent process Use the kill command and send the SIGCHLD signal. This might trigger the required wait() function and return things in a normal state. kill -SIGCHLD PID-OF-PARENT-PROCESS\nOption 2: kill the parent process If the first option does not work, then try stopping the parent process. Sometimes this might properly close the child process as well.\nOption 3: reboot Both options not working? Then a reboot will, as this will stop everything and start the system with a fresh process table.\n","permalink":"https://linux-audit.com/processes/faq/how-to-kill-a-zombie-process/","tags":["faq","howto","linux","processes"],"title":"How to kill a zombie process"},{"categories":["System Administration"],"contents":"To find running processes by their name and PID , use the pgrep command. As the name implies, it is like the grep command, but for processes.\nExamples using pgrep To only see the PID (or PIDs), use pgrep followed by the process name.\n# pgrep nginx 51055 60297 60298 If you want to see both the PID and process name, add the --list-name option.\n# pgrep --list-name nginx 51055 nginx 60297 nginx 60298 nginx This output can also be useful to confirm the right processes are targeted when using the pkill command.\n","permalink":"https://linux-audit.com/processes/faq/how-to-show-running-process-with-pid/","tags":["faq","howto","linux","pgrep","processes"],"title":"How to show a running process name and its process ID (PID)"},{"categories":["System Administration"],"contents":"To get the PID of a process, use the pidof command. If there are multiple processes with the same name, then all PIDs will be displayed.\nExamples using pidof Run pidof followed by the process name that you want to search for. If there is a match, the command will be display the related process ID or IDs.\n# pidof nginx 60013 60012 51055 To get only one PID returned, use the pidof command with the -s option.\n# pidof -s systemd 60013 Relevant FAQ: How to kill a running process by its name?\n","permalink":"https://linux-audit.com/processes/faq/how-to-find-process-id-by-its-process-name/","tags":["faq","howto","linux","processes"],"title":"How to find all process IDs by its process name"},{"categories":["System Administration"],"contents":"Linux uses signals to interact and define the state of a process. It uses POSIX reliable and real-time signals. The first are considered standard signals.\nMany programs are build using glibc and therefore use functions like kill(2) to send a signal to a process or processes group, or even all processes on the system. A process can decide to ignore a signal or take an action after it is received by a signal handler, a routine to catch incoming signals.\nSignal list Signal name Numeric value Description SIGHUP 1 Signal to tell user\u0026rsquo;s terminal is disconnected. For some processes it reloads configuration SIGINT 2 Interrupt, for example when using CTRL+C, usually with proper clean up of system resources, such as temporary files SIGQUIT 3 Like SIGINT, but usually with CTRL+, often not doing clean up of resources SIGILL 4 Illegal instruction, process performs garbage execution or privileged instruction SIGTRAP 5 Signal used by debuggers SIGABRT 6 Process called abort() function, deliberate crash SIGIOT 6 Generated by PDP-11 \u0026ldquo;iot\u0026rdquo; instruction, on Linux SIGABRT is used SIGBUS 7 Like SIGSEGV, but when trying to use invalid memory address SIGEMT - Emulator trap, received when performing certain unimplemented instructions SIGFPE 8 Floating-point exception, but also occurs with fatal arithmetic errors like division by zero or overflow SIGKILL 9 Forced stop of a process, more forceful than SIGTERM SIGUSR1 10 Reserved for a developer to use and define a relevant action SIGSEGV 11 Segmentation fault or access violation, usually when incorrect memory location is attempted to access. SIGUSR2 12 Similar to SIGUSR1, second reserved signal SIGPIPE 13 Broken pipe, related to pipes and FIFO special files SIGALRM 14 Expiration of timer that measures real or clock time, used by function like alarm() SIGTERM 15 Tell process to stop SIGSTKFLT 16 Stack fault, sent to process when a stack overflow or stack underflow occurs SIGCHLD 17 Signal sent to parent process when child process is stopped SIGCLD - Obsolete, replaced by SIGCHLD SIGCONT 18 SIGSTOP 19 Stop a process, can not be handled nor ignored by a process SIGTSTP 20 Interactive stop request, can be ignored by a process SIGTTIN 21 Signal to instruct that reading from terminal is not possible, for example for tasks running in background SIGTTOU 22 Same as SIGTTOU, but for writing output to terminal SIGURG 23 Urgent signal for out-of-band data, special handling SIGXCPU 24 CPU time limit exceeded SIGXFSZ 25 File size limit exceeded, such as a defined soft limit SIGVTALRM 26 Short for virtual time alarm, expiration of timer that measures CPU time by the current process SIGPROF 27 Used for code profiling, CPU time used by process and CPU time expended on behalf of process by the system itself SIGWINCH 28 Signal used for events related to resizing of window SIGIO 29 Signal to inform when a file descriptor is ready, for example to perform input or output SIGPOLL - System V signal, very similar to SIGIO SIGPWR 30 Signal only used by init process, typically due to hardware issue SIGINFO - Information request, may let the process share some information such as its status SIGLOST - Resource lost, such as a lock on NFS resource SIGSYS 31 Bad argument provided to a system call (syscall) SIGUNUSED 31 ","permalink":"https://linux-audit.com/processes/linux-process-signals/","tags":["linux","processes"],"title":"Linux process signals and their meaning"},{"categories":["System Administration"],"contents":"To find running processes by their name and stop it, use the killall or pkill command.\nExamples using killall The killall command has the --interactive option that will ask a confirmation before stopping processes.\n# killall -i nginx Kill nginx(51055) ? (y/N) n Kill nginx(60297) ? (y/N) n Kill nginx(60298) ? (y/N) n nginx: no process found As we answered \u0026rsquo;n\u0026rsquo; to all questions, killall won\u0026rsquo;t stop any running processes.\nExamples using pkill pkill nginx\nWhen using pkill, there is no interactive confirmation. Use the pgrep command with the -l option to confirm the right processes are targeted.\nRelevant FAQ: How to show a running process name and its process ID (PID)?\nProcess signals Most of the tools that can send a signal to a process or a group of processes, can be instructed to send a specific process signal. This might be useful when a process is stuck and does not respond to the default SIGTERM signal.\n","permalink":"https://linux-audit.com/processes/faq/how-to-kill-a-running-process-by-its-name/","tags":["faq","howto","kill","linux","pgrep","processes"],"title":"How to kill a running process by its name"},{"categories":["Kernel"],"contents":"This sysctl key defines if a system should allow forwarding of IPv4 network packets. This functionality is required for systems that act as a gateway or router.\nIP forwarding is normally not required for most desktops and servers.\n","permalink":"https://linux-audit.com/kernel/sysctl/net/net.ipv4.ip_forward/","tags":["kernel","kernel hardening","linux","network","sysctl"],"title":"Sysctl: net.ipv4.ip_forward"},{"categories":["Network","System Administration"],"contents":"To see information about your IP address, use the ip command with the address subcommand. Use the shortened version ip a for quick access.\nOutput of ip The ip command can show IP addresses, but will also include information like your MAC address, subnet, MTU, and status.\n# ip address 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens18: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether ab:cd:de:12:34:56 brd ff:ff:ff:ff:ff:ff altname enp0s18 inet 192.168.1.100/24 brd 192.168.2.255 scope global ens18 valid_lft forever preferred_lft forever inet6 fe80::be24:11ff:abde:1234/64 scope link valid_lft forever preferred_lft forever In this case the interface name is ens18 and the IP address is 192.168.1.100.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-network-ip-address/","tags":["dns","faq","howto","linux","network"],"title":"How to see the the network IP address of your system"},{"categories":["Network","System Administration"],"contents":"There are multiple ways to find out your internet IP address from the command line. One way is using the dig command.\nUsing dig With dig we can query the hostname myip.opendns.com. To prevent any intermediate DNS resolver to send an incorrect answer, use the source system.\ndig +short myip.opendns.com @resolver1.opendns.com\nUsing curl Some external websites also provide the IP address. If there is no proxy between you and the related server, it can also provide your IP address.\ncurl ifconfig.me\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-ip-address-of-internet-connection-from-command-line/","tags":["dns","faq","howto","linux","network"],"title":"How to see the IP address of your internet connection"},{"categories":["Network","System Administration"],"contents":"Systems using systemd have the resolvectl command to show the configured DNS servers. It also show the server that is currently active.\nShow active DNS configuration Run the resolvectl command without any options to see the status. The configured DNS servers will be displayed, including the one that is currently being used.\n# resolvectl Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub Link 2 (ens18) Current Scopes: DNS Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 192.168.1.10 DNS Servers: 192.168.1.10 192.168.1.11 DNS Domain: internal.example.com Not using systemd? When systemd is not being used, then typically the first DNS server configured in the file /etc/resolv.conf is used for processing the DNS requests.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-which-dns-server-is-used/","tags":["dns","faq","howto","linux","network"],"title":"How to see which DNS server is used"},{"categories":["System Administration"],"contents":"If you are looking for files or their permissions, then the find command has you covered. Let\u0026rsquo;s have a look how to find files that are writable by a user, group, or others.\nFind all writable files We can use the -perm option to see what files have a specific set of file permissions. To match one of the three selectors (user, group, other), we can use a slash.\nfind . -perm /222\nThis command will search in the current directory and any file that is writable, will show up.\nWritable files by others Want to find only the files that are writable by others? Then set the first two selector to zero, so that the matching happens on the \u0026lsquo;other\u0026rsquo; only.\nfind . -perm /002\nThe alternative notation for this command is:\nfind . -perm /o=w\nThis type of notation might be easier to read. Want to combine? Sure!\nfind . -perm /g=w,o=w\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-find-writable-files/","tags":["faq","file system","howto","linux"],"title":"How to find writable files"},{"categories":["System administration"],"contents":"The apt-file command shows information about packages and the available files. It can search both in the package name and provided files, even if a package is not installed.\n","permalink":"https://linux-audit.com/system-administration/commands/apt-file/","tags":["linux","monitoring","packages","package manager"],"title":"apt-file: show information about packages and related files"},{"categories":["Cheat sheets","System Administration"],"contents":"This cheat sheet helps becoming more fluent with the apt command, the package manager used on Linux systems running Debian, Ubuntu, and others.\nBasic usage These are common subcommands to be used together with apt:\nCommand Intended action autoremove Clean up packages that are no longer needed install PACKAGE Install package search PATTERN Search for available package according to the pattern purge PACKAGE Remove the remainders after removing a package remove PACKAGE Remove package, leave custom configurations show PACKAGE Provide information about a package update Refresh the repository data upgrade Pull in and install upgrades for packages that are installed Common options include:\nLong Short Action --download-only -d Just download a package, do not install --quiet -q Silent operation --simulate -s Perform simulation, no actual actions --yes -y Perform \u0026lsquo;yes\u0026rsquo; to prompts, automating actions Information about packages See which packages are available that start with \u0026rsquo;nginx\u0026rsquo; with the list subcommand and really quiet output.\n# apt -qq list nginx* nginx-common/jammy-updates 1.18.0-6ubuntu14.4 all nginx-confgen/jammy 2.0-1 amd64 nginx-core/jammy-updates 1.18.0-6ubuntu14.4 amd64 nginx-doc/jammy-updates 1.18.0-6ubuntu14.4 all nginx-extras/jammy-updates 1.18.0-6ubuntu14.4 amd64 nginx-full/jammy-updates 1.18.0-6ubuntu14.4 amd64 nginx-light/jammy-updates 1.18.0-6ubuntu14.4 amd64 nginx/jammy-updates 1.18.0-6ubuntu14.4 amd64 Show information of a single package to learn what it does, its dependencies, size, and other possible relevant information.\napt show nginx\nInstalled packages See all installed packages with the list subcommand and --installed option.\napt list --installed\nInstallation of packages Installation happens with the subcommand install followed by the package.\napt install nginx\nInstall a local .deb file is also possible.\napt install myfile.deb\nUpgrading packages Refresh the repository to pull in the latest available information of the underlying databases.\napt update\nSee packages that have a newer version (upgrade):\napt list --upgradable\nInstall available upgrades:\napt upgrade\nRelated tools Use apt-file to see to which package a file belongs or what files are provided by a package.\n","permalink":"https://linux-audit.com/cheat-sheets/apt/","tags":["apt","cheatsheet","debian","howto","linux","package manager","ubuntu"],"title":"apt cheat sheet"},{"categories":["File systems","System Administration"],"contents":"Use the du command to scan a directory and find when the last modification was made. The related modification time will be displayed when using du with the --time option.\n# du -sh --time /var 1.2G\t2024-06-01 21:45\t/var The time option will show the last modification time of a file or subdirectory within the directory that was specified.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-find-the-last-modification-within-a-directory/","tags":["faq","file system","howto","linux"],"title":"How to find when the last modification happened in a directory"},{"categories":["File systems","System Administration"],"contents":"Use the du command we can count the directory size. By using it with the options --human-readable and --summarize, a summarized total will be displayed.\nDefine a threshold To see files bigger than 20 kilobytes, define the threshold.\ndu -h --threshold=20K\nTo see files smaller than the defined threshold, add a minus before the number.\nLooking in multiple directories Another option is using find and define the minimum size of a file. For example all files that are at least 1 megabyte:\nfind . -size +1M\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-files-greater-than-specific-file-size/","tags":["faq","file system","howto","linux"],"title":"How to see files greater than a specific size"},{"categories":["File systems","System Administration"],"contents":"Use the ls command with -a combined to see hidden files on Linux. For a friendlier output, combine it with the option -l.\nHidden files Hidden files start with a dot and are typically not displayed by default when using ls. Use \u0026lsquo;-al\u0026rsquo; to show them.\n# ls -al total 90 drwxrwxr-x 7 michael michael 12 Apr 8 07:34 . drwxrwxr-x 3 michael michael 3 Apr 28 11:19 .. -rwxrwxr-x 1 michael michael 447 Mar 21 14:09 .editorconfig -rwxrwxr-x 1 michael michael 17 Mar 21 14:09 .gitignore -rwxrwxr-x 1 michael michael 1080 Mar 21 14:09 LICENSE.md -rwxrwxr-x 1 michael michael 6923 Mar 21 14:09 README.md ","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-hidden-files/","tags":["faq","file system","howto","linux"],"title":"How to see hidden files"},{"categories":["File systems","System Administration"],"contents":"With the du command the directory size can be retrieved. By using it with the options --human-readable and --summarize, a summarized total will be displayed.\nDisk Usage The du tool is very powerful to show disk usage of files and directories, but it needs the right options to get the right information. A good combination to start is using \u0026lsquo;-h\u0026rsquo;, so the output becomes human-readable and the output size is shown in kilobytes, megabytes, etc.\ndu -h\nNow the output can be long, so summarizing is a good idea.\ndu -hs\nYou can also define a specific path\ndu -hs /etc\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-directory-size/","tags":["faq","file system","howto","linux"],"title":"How to see the size of a directory"},{"categories":["File systems","Linux forensics","System Administration"],"contents":"Use the df command with the --inodes to see inode usage on a file system.\nShow inode usage The output of df --inodes will include the total number of inodes (Inodes), how many there are in use (IUsed), and remaining ones (IFree). For easier understanding its usage, the inode usage will also be listed in a percentage (IUse).\n# df --inodes Filesystem Inodes IUsed IFree IUse% Mounted on tmpfs 500759 765 499994 1% /run /dev/mapper/ubuntu--vg-ubuntu--lv 933888 144120 789768 16% / tmpfs 500759 4 500755 1% /dev/shm tmpfs 500759 3 500756 1% /run/lock /dev/sda2 114688 320 114368 1% /boot tmpfs 100151 25 100126 1% /run/user/1000 192.168.1.1:/projects 15283468 31186 15252282 1% /mnt/projects To shorten this command, combine the options and use df -i.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-inode-usage/","tags":["faq","file system","howto","inode","linux"],"title":"How to see inode usage"},{"categories":["File systems","Linux forensics","System Administration"],"contents":"Use the df command to see disk space usage. By default, the output will show 1K-blocks. To see sizes in a human-readable format, use the --human-readable option.\nShow all file systems and their disk usage By specifying the --all option, we see all mounted file systems, even though they don\u0026rsquo;t have disk usage information.\n# df --all --human-readable Filesystem Size Used Avail Use% Mounted on sysfs 0 0 0 - /sys proc 0 0 0 - /proc udev 1.9G 0 1.9G 0% /dev devpts 0 0 0 - /dev/pts tmpfs 392M 1.1M 391M 1% /run /dev/mapper/ubuntu--vg-ubuntu--lv 14G 6.9G 6.4G 53% / securityfs 0 0 0 - /sys/kernel/security tmpfs 2.0G 12K 2.0G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock cgroup2 0 0 0 - /sys/fs/cgroup pstore 0 0 0 - /sys/fs/pstore bpf 0 0 0 - /sys/fs/bpf systemd-1 - - - - /proc/sys/fs/binfmt_misc hugetlbfs 0 0 0 - /dev/hugepages mqueue 0 0 0 - /dev/mqueue debugfs 0 0 0 - /sys/kernel/debug tracefs 0 0 0 - /sys/kernel/tracing configfs 0 0 0 - /sys/kernel/config none 0 0 0 - /run/credentials/systemd-sysusers.service fusectl 0 0 0 - /sys/fs/fuse/connections systemd-1 - - - - /mnt/projects /dev/loop2 39M 39M 0 100% /snap/snapd/21465 /dev/loop0 88M 88M 0 100% /snap/lxd/28373 /dev/loop4 88M 88M 0 100% /snap/lxd/27948 /dev/loop5 64M 64M 0 100% /snap/core20/2264 /dev/sda2 1.7G 253M 1.4G 16% /boot binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /run/rpc_pipefs tmpfs 392M 1.1M 391M 1% /run/snapd/ns nsfs 0 0 0 - /run/snapd/ns/lxd.mnt tmpfs 392M 4.0K 392M 1% /run/user/1000 /dev/loop6 64M 64M 0 100% /snap/core20/2318 /dev/loop3 39M 39M 0 100% /snap/snapd/21759 192.168.1.10:/Projects 10G 2.8G 7.3G 28% /mnt/projects To shorten this command, combine the options and use df -ah.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-used-and-free-disk-space/","tags":["faq","file system","howto","linux"],"title":"How to see used and free disk space"},{"categories":["File systems","Linux forensics","System Administration"],"contents":"The find command has so many options, that it may be overwhelming to learn all it has to offer. Let\u0026rsquo;s have a look at this particular question.\nFinding symbolic links The obvious way to find a symbolic link is define -typel. This will search and show all symbolic links. To find all symbolic links that point to a directory, use find with the -type l and combine it with the -xtyped options.\nfind . -type l -xtype d To search in a specific directory, replace the dot with the directory to start.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-find-symbolic-links-that-point-to-a-directory/","tags":["faq","file system","forensics","howto"],"title":"How to find symbolic links that point to a directory"},{"categories":["File systems","Linux forensics","System Administration"],"contents":"The diff command is great for discovering differences between two files. What you might not expect from it, is its ability to compare two directories!\nCompare two directories To see the differences between two directories, we can use diff with the --brief and --recursive options. By providing two directories, this will search in them and highlight what files are only in one of them. If it is available in both, it will compare to see if the file is the same or also differs.\n# diff --brief --recursive test1 test2 Files test1/readme.md and test/readme.md differ Only in test2: file-permissions.md Only in test1: symbolic-links.md Only in test1: biggest-directories.md Only in test2: linux-file-permissions.md Without the brief output we would also see the exact differences between files. Comparing two directories using this command is amazing, right?\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-compare-two-directories-and-find-the-differences/","tags":["faq","file system","howto"],"title":"How to compare two directories and find the differences"},{"categories":["Hardware"],"contents":"With the help of usbmon we have a cost-effective solution to monitor USB communications between the kernel and a hardware device. It is an alternative to hardware USB protocol analyzers for those on a budget or having to do it once.\nLoading of usbmon kernel module The first step is to load the related kernel module. This can be done using the modprobe command.\n$ sudo modprobe usbmon Validate if the usbmon kernel module is loaded using the lsmod command.\n$ sudo lsmod | grep usbmon usbmon 45056 0 If everything looks good, there should be a line visible of our loaded module.\nConfirm USB device identification Next step is to look if the hardware device is listed correctly. The output of the lsusb command will show all connected USB devices to the system. For example, a snipped output could look like this:\n$ lsusb Bus 003 Device 009: ID 0fd9:0084 Elgato Systems GmbH Stream Deck Plus \u0026lt;output snipped\u0026gt; In this case, we are interested on device 009 on bus 003. The bus defines our interface name, in this case 3. The interface is named usbmon followed by the bus number, so usbmon3. This is needed for the next step.\nCapture traffic With the bus number we could discover the interface. In this case, usbmon3. This is something that we can monitor using TShark or Wireshark if you prefer the graphical user interface.\ntshark -i usbmon3\nNext thing to do is monitor the traffic on the bus and see what is related to the device. If you are unsure about the identifiers, have a look at the output of lsusb -t -v.\n","permalink":"https://linux-audit.com/hardware/usb/monitoring-usb-communications-using-usbmon-interface/","tags":["hardware","linux","lsmod","usb"],"title":"Monitoring USB communications using usbmon interface"},{"categories":["System Administration"],"contents":"The file /etc/ssh/sshd_config is the main configuration file of the OpenSSH server daemon (sshd).\nIt defines the primary set of settings, with the option of being overridden by configuration files from /etc/ssh/sshd_config.d/*.conf.\nTo learn more about the available configuration settings, have a look at the section OpenSSH server configuration.\n","permalink":"https://linux-audit.com/system-administration/files/etc-ssh-sshd_config/","tags":["configuration","linux","ssh","sshd","sshd_config"],"title":"/etc/ssh/sshd_config: SSH daemon configuration"},{"categories":["System Administration"],"contents":"Introduction Systemd timers are the equivalent of cron and cronjobs. For modern Linux systems they most likely are a full replacement, including additional features.\nTimers in systemd can be recognized by their extension .timer and are plaintext files that include their configuration. They describe what should happen, when, and if there are specific conditions that needs to be met. It may include additional information like documentation and comments.\nOne important thing to know upfront when using timers is that a timer does not come alone. It is a trigger for a service unit (.service) and therefore should have the same name. The only difference in the name is obviously the file extension. If you are new to systemd units, have a look at systemd units and their purpose.\nAdditional benefits of timers Timers have a few additional benefits that typically does not exist in cron or requires additional work.\nDetailed management Systemd units and their status are tracked carefully. Details such as activation and last execution are stored. This makes it easy to see when a task was performed. In the case of timers, we can also see when it will be performed in the near future. With cron this is harder to see, especially when jobs are divided over multiple locations (crontab, cron.hourly, cron.daily, cron.weekly, etc).\nDelayed start after boot One of the options is to start a task just after the system is booted, like updating software packages. But as the system will be very busy just after the start, it might be useful to wait for a few minutes. This can be achieved using the OnBootSec setting, part of the Timer section.\nSetting: OnBootSec Value: time Section: Timer Example: OnBootSec=5min Delayed start with random time When having many systems performing a task at the the same hour and minute, it can cause unneeded strain on the network, internet connection, or receiving systems. Timers can solve this issue by delaying the start time of a task with a random interval. The setting is RandomizedDelaySec and part of the Timer section.\nWhile the setting would indicate the time is to be set in seconds, a time can also be specified, like 12h. This way the task will be delayed between 0 seconds and half a day.\nSetting: RandomizedDelaySec Value: time Section: Timer Example: RandomizedDelaySec=12h Delay after unit activation Sometimes you want to create a timer, but not directly active it. Maybe there is not enough data to process, or you want to wait till the next day. By defining OnUnitActiveSec in the Timer section, we can delay execution as well.\nSetting: OnUnitActiveSec Value: time Section: Timer Example: OnUnitActiveSec=24h See available systemd timers On a regular Linux distribution there will be timers defined by the distribution itself and as part of common software packages. Use the subcommand list-timers to show the existing timers.\n# systemctl list-timers NEXT LEFT LAST PASSED UNIT ACTIVATES Fri 2024-05-24 10:10:35 UTC 2h 27min left Thu 2024-05-23 04:56:31 UTC 1 day 2h ago man-db.timer man-db.service Fri 2024-05-24 11:48:14 UTC 4h 5min left Thu 2024-05-23 19:57:03 UTC 11h ago fwupd-refresh.timer fwupd-refresh.service Fri 2024-05-24 16:01:53 UTC 8h left Thu 2024-05-23 19:56:56 UTC 11h ago apt-daily.timer apt-daily.service Fri 2024-05-24 20:10:39 UTC 12h left Fri 2024-05-24 04:47:52 UTC 2h 55min ago motd-news.timer motd-news.service Sat 2024-05-25 00:00:00 UTC 16h left Fri 2024-05-24 00:00:21 UTC 7h ago dpkg-db-backup.timer dpkg-db-backup.service Sat 2024-05-25 00:00:00 UTC 16h left Fri 2024-05-24 00:00:21 UTC 7h ago logrotate.timer logrotate.service Sat 2024-05-25 00:05:14 UTC 16h left Fri 2024-05-24 00:05:14 UTC 7h ago update-notifier-download.timer update-notifier-download.service Sat 2024-05-25 00:16:14 UTC 16h left Fri 2024-05-24 00:16:14 UTC 7h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Sat 2024-05-25 06:06:59 UTC 22h left Fri 2024-05-24 06:50:34 UTC 52min ago apt-daily-upgrade.timer apt-daily-upgrade.service Sun 2024-05-26 03:10:58 UTC 1 day 19h left Sun 2024-05-19 03:11:03 UTC 5 days ago e2scrub_all.timer e2scrub_all.service Mon 2024-05-27 01:04:09 UTC 2 days left Mon 2024-05-20 00:01:32 UTC 4 days ago fstrim.timer fstrim.service Mon 2024-05-27 07:15:58 UTC 2 days left Thu 2024-05-23 19:56:56 UTC 11h ago update-notifier-motd.timer update-notifier-motd.service 12 timers listed. Pass --all to see loaded but inactive timers, too. Besides the timing (next, left, last, passed), we can see the unit name, but also the column ACTIVATES.\nHow to see the configuration of a timer In the column UNIT, when using list-timers, the file name is shown. This file name is not the full path, so it can be stored in a few common locations. Fortunately, you don\u0026rsquo;t have to know where it is located (yet). With the subcommand cat we can easily see the content of the timer file and its location.\n# systemctl cat man-db.timer # /lib/systemd/system/man-db.timer [Unit] Description=Daily man-db regeneration Documentation=man:mandb(8) [Timer] OnCalendar=daily RandomizedDelaySec=12h Persistent=true [Install] WantedBy=timers.target When using systemctl cat, the first line is displayed as a comment and refers to the full path of the unit file.\nStatus of a timer To learn more about a particular timer and its status, we can query it using the status subcommand. Similar to a service, this will give us more insights, but tailored to the timer itself.\n# systemctl status fstrim.timer ● fstrim.timer - Discard unused blocks once a week Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled) Active: active (waiting) since Wed 2024-05-08 23:56:25 UTC; 2 weeks 1 day ago Trigger: Mon 2024-05-27 01:04:09 UTC; 2 days left Triggers: ● fstrim.service Docs: man:fstrim May 08 23:56:25 webdev01 systemd[1]: Started Discard unused blocks once a week. In this example, we can see that is is active and in 2 days the timer will be triggered. The Triggers line shows us the relevant service (fstrim.service).\nTimer types and schedule Timers are scheduled by their type. A monotonic timer is activated based on another event, such as activation of the unit or boot time. Examples of this type were already covered in the benefits above.\nAnother type of a real-time timer. It is planned by using a calendar event, like the day name (Monday) or specific hour of the day. This type is commonly used to run tasks on a repeating rate.\nBy day When using the OnCalendar option, we can schedule a task daily, exactly at the hour/minute/second mark.\nOnCalendar=Mon..Sun *-*-* 4:00:00 Another option is defining multiple ranges.\nOnCalendar=Mon..Fri *-*-* 22:00 OnCalendar=Sat,Sun *-*-* 14:00 Multiple hours is also possible. To run a task daily at 6 AM and 6 PM, define the entry without a day and only the hours, separated by a comma.\nOnCalendar=*-*-* 6,18:00 To repeat a task multiple times an hour, we can define this using a slash. For example, having the systemd timer run every 5 minutes.\nOnCalendar=*-*-* *:0/5:00 Weekly The easiest way to plan a task weekly is by using the weekly value.\nOnCalendar=weekly An alternative is to define the specific day and hour.\nOnCalendar=Sun *-*-* 06:00:00 By specific day of the month It is also possible to plan a task at the first of the month, or a specific month of the year.\nOnCalendar=*-*-1 00:01 In March, daily? Set the month\nOnCalendar=*-3-* 00:01 Only on Sunday if it is an even month.\nOnCalendar=Sun *-2,4,6,8,10,12-* 00:01 As you can see, this system of scheduling is very flexible.\nTesting your timer Due to its flexibility, one might make an error. To learn when the next planned action would happen, we can use the systemd-analyze command with its subcommand calendar.\n# systemd-analyze calendar \u0026#34;Sun *-6,8-* 00:01\u0026#34; Original form: Sun *-6,8-* 00:01 Normalized form: Sun *-06,08-* 00:01:00 Next elapse: Sun 2024-06-02 00:01:00 UTC From now: 1 week 1 day left Systemd timers commands Command Goal systemctl cat UNIT.timer Show the content of a timer systemctl enable UNIT.time Enable the timer systemctl list-timers Show active timers systemctl list-timers PATTERN Show active timers that match a pattern (e.g. log*) systemctl list-timers --all Show all timers, including inactive timers systemctl start UNIT.timer Start the timer systemctl status UNIT.timer Show status of the timer systemctl stop UNIT.timer Stop the timer Converting from cron to systemd timers Need to convert entries from cron to systemd? Here are a few common ones.\nCron systemd timer hourly OnCalendar=--* *:00:00 daily OnCalendar=--* 00:00:00 weekly OnCalendar=Sun --* 00:00:00 monthly OnCalendar=--01 00:00:00 ","permalink":"https://linux-audit.com/systemd/systemd-timers/","tags":["systemd","systemd-analyze"],"title":"Systemd timers"},{"categories":["Network","System Administration"],"contents":"The ss command is probably the best tool to query statistics for network connections, including the total number of open connections.\nShow active connections With --summary the active state can be shown, including established TCP connections.\n# ss --summary Total: 225 TCP: 26 (estab 10, closed 10, orphaned 0, timewait 10) Transport Total IP IPv6 RAW\t1 0 1 UDP\t3 2 1 TCP\t16 14 2 INET\t20 16 4 FRAG\t0 0 0 This output is slim, but provides a first good insight on how busy the system is when it comes to network connectivity. Great for troubleshooting servers like a web server.\nWant to see active ongoing connections and monitor them? Consider using the iftop command.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-number-of-open-connections/","tags":["faq","howto","linux","network"],"title":"How to see the number of open connections on Linux"},{"categories":["System Administration"],"contents":"When a process is started, the kernel does some fine accounting to document the related details, such as start time, command parameters, and environment variables. This information we can use when doing an investigation or learn more about a particular process.\nStart time of a process One option to query when a particular process was started, is using the ps command and the related process ID (PID). The PID can be seen when running this command with common options like -ef. Another option is using pidof or the pgrep commands.\nWhen we have the PID, we can query the start time.\n# ps -o cmd,lstart -p 1 CMD STARTED /sbin/init Wed May 8 23:56:05 2024 Start time of a systemd unit or service If the system is using systemd as its service manager, then we can ask systemctl to provide these details. The show subcommand combined with a specific property \u0026lsquo;ExecMainStartTimestamp\u0026rsquo; will reveal the date and time that the process was started.\n# systemctl show --property ExecMainStartTimestamp --value ssh.service Wed 2024-05-08 23:56:26 UTC How long ago was a process started? For shell scripting purposes it might be useful to learn how long ago a process was started, or the elapsed time in seconds. To get this number, we can use the \u0026rsquo;etimes\u0026rsquo; column.\n# ps -o etimes= -p 1` 1159170 The outputted number will obviously increase if you repeat the command.\nGot other useful commands to query when a process was started?\n","permalink":"https://linux-audit.com/system-administration/faq/how-to-see-how-long-a-process-is-running/","tags":["faq","howto","linux","processes","ps"],"title":"How to see when a process was started"},{"categories":["System Administration"],"contents":"To know when a system was started, we want to query its uptime, or the time since booting up. With the uptime command, we can easily retrieve this information. Let\u0026rsquo;s have a look!\nSystem uptime To request the uptime, simply run the command without any parameters.\n# uptime 08:26:05 up 13 days, 8:29, 4 users, load average: 0.06, 0.02, 0.00 This output gives a good impression, but is not very precise. To zoom in to a specific date and time, use the --since option.\n# uptime --since 2024-05-08 23:56:06 This output is easier to read and tells us the exact time when the system was started.\nQuery init process details We can also query the start time of the first process, which is normally the init system.\nps -o lstart= -p 1 Wed May 8 23:56:05 2024 As expected, this shows a similar time as our uptime output above. By using -o we can select the columns that we are interested in. The \u0026lsquo;=\u0026rsquo; in the column name means the related header will be hidden.\nUsing systemd to determine uptime If the system is using systemd, we can also use systemctl to query one of the first targets. We can see the start time as an date, but also in seconds.\n# systemctl show --property ActiveEnterTimestamp init.scope ActiveEnterTimestamp=Wed 2024-05-08 23:56:09 UTC Most likely this will have a slightly different time as the other commands, as this is a recorded time of a state change within the related systemd unit (scope).\nAlso the journal will reveal the boot time.\n# journalctl --list-boots | tail -1 0 7ec0e89c1de74b71bcc9e004a7f30d45 Wed 2024-05-08 23:56:09 UTC—Wed 2024-05-22 09:45:01 UTC An alternative is to select the current boot (id 0) and pull in the first line. journalctl --boot=0 | head -1\nGot other options to query the uptime of a system?\n","permalink":"https://linux-audit.com/system-administration/faq/how-to-see-when-system-was-started/","tags":["faq","howto","linux","uptime"],"title":"How to see when the system was started (uptime)"},{"categories":["System administration"],"contents":"","permalink":"https://linux-audit.com/system-administration/commands/smem/","tags":["forensics","linux","monitoring","memory","processes","swap"],"title":"smem"},{"categories":["System administration"],"contents":"The iftop command is a Linux tool to see the bandwidth by host. It listens to network traffic on the specified network interface or the first one it can find. It needs root access to be able to listen to this traffic.\nIn most cases it is advised to run iftop with the -n option to reduce additional traffic caused by DNS , which translates all IP addresses to hostnames.\nIftop can be instructed to monitor the traffic per network with the -F option, which might be useful to see traffic going to specific networks.\nThe filters that iftop uses are those from pcap, a packet capture library. The syntax is there the same as tools like tcpdump.\n","permalink":"https://linux-audit.com/system-administration/commands/iftop/","tags":["linux","monitoring","network","troubleshooting"],"title":"iftop"},{"categories":["Network","System Administration"],"contents":"The iftop command is one of the tools that is great for shows active connections and bandwidth usage on a Linux system. As the name implies, it is like \u0026rsquo;top\u0026rsquo; but for network connections.\nShow active connections iftop\nBy default most options are enabled. This may be useful, but also confusing if you are looking for specific traffic. A few good shortcuts include n (hostnames), N (service names).\n224.0.0.251:5353 \u0026lt;= 192.168.1.16:5353 604B 2,36Kb 483b 121b 192.168.1.25:40238 \u0026lt;= 12.20.9.18:443 31,1KB 444b 134b 67b 224.0.0.251:5353 \u0026lt;= 192.168.1.34:5353 142B 568b 114b 28b 192.168.1.25:54226 \u0026lt;= 192.168.1.170:8443 96,5KB 0b 83b 19,2Kb 192.168.1.25:54228 \u0026lt;= 192.168.1.170:8443 96,8KB 0b 83b 95b 255.255.255.255:10001 \u0026lt;= 192.168.1.1:35558 9,56KB 0b 51b 38b 192.168.1.25:47184 \u0026lt;= 192.168.1.170:8443 573B 0b 42b 115b 192.168.1.25:47196 \u0026lt;= 192.168.1.170:8443 573B 0b 42b 115b 255.255.255.255:10001 \u0026lt;= 192.168.1.9:60489 4,78KB 0b 26b 19b 192.168.1.25:50982 \u0026lt;= 192.168.1.170:22 22,7KB 0b 0b 4,00Kb 192.168.1.25:47736 \u0026lt;= 192.168.1.170:22 12,4KB 0b 0b 2,49Kb 192.168.1.25:33012 \u0026lt;= 5.6.2.140:443 6,42KB 0b 0b 1,28Kb TX: cum: 1,70MB peak: 205Kb rates: 244b 344b 16,3Kb RX: 5,98MB 450Kb 3,35Kb 1,03Kb 29,5Kb TOTAL: 7,68MB 495Kb 3,59Kb 1,37Kb 45,9Kb Bandwidth usage To better see which connection is causing a lot of traffic, enable the bars using the b key. Depending on how traffic is distributed, it may be useful to switch between a linear or logarithmic scale with the L key.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-active-connections-and-bandwidth-usage/","tags":["faq","howto","linux","network"],"title":"How to see active connections and bandwidth usage on Linux"},{"categories":["System administration"],"contents":"","permalink":"https://linux-audit.com/system-administration/commands/pidstat/","tags":["cpu","linux","memory","monitoring","troubleshooting"],"title":"pidstat"},{"categories":["System administration","Troubleshooting"],"contents":"Got a busy system that comes to a halt due it being too busy? In this article we look at troubleshooting issues related to CPU usage.\nMonitoring CPU usage The tool top might be the most familiar tool to monitor CPU or memory usage. A good alternative is the pidstat tool. It can be using an interval and easily show active processes, followed by a summary.\n# pidstat 3 Linux 6.5.0-28-generic (workstation) 20-05-24 _x86_64_\t(8 CPU) 13:45:47 UID PID %usr %system %guest %wait %CPU CPU Command 13:45:50 0 638 0,00 0,66 0,00 0,00 0,66 3 irq/204-nvidia 13:45:50 1000 1744 0,33 1,00 0,00 0,00 1,33 6 pulseaudio 13:45:50 1000 2050 1,66 0,33 0,00 0,00 1,99 2 gnome-shell 13:45:50 1000 3767 1,99 1,33 0,00 0,00 3,32 3 firefox 13:45:50 1000 3985 1,00 0,66 0,00 0,00 1,66 7 Isolated Web Co 13:45:50 1000 4277 0,33 0,00 0,00 0,00 0,33 5 WebExtensions 13:45:50 1000 25736 0,00 0,33 0,00 0,00 0,33 5 Isolated Web Co 13:45:50 1000 560859 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co 13:45:50 1000 657165 0,33 0,00 0,00 0,00 0,33 3 Isolated Web Co 13:45:50 1000 858923 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co 13:45:50 1000 1235407 0,33 0,00 0,00 0,00 0,33 1 Isolated Web Co 13:45:50 0 1284255 0,00 0,33 0,00 0,00 0,33 7 kworker/7:1-events 13:45:50 0 1284904 0,00 0,33 0,00 0,00 0,33 0 kworker/0:2-pm 13:45:50 0 1285798 0,00 0,33 0,00 0,00 0,33 3 kworker/3:0-events 13:45:50 1000 1286455 1,00 0,00 0,00 0,00 1,00 0 Isolated Web Co 13:45:50 0 1287603 0,00 0,33 0,00 0,00 0,33 0 kworker/0:0-events 13:45:50 UID PID %usr %system %guest %wait %CPU CPU Command 13:45:53 0 638 0,00 0,67 0,00 0,00 0,67 3 irq/204-nvidia 13:45:53 1000 1744 0,00 1,00 0,00 0,00 1,00 6 pulseaudio 13:45:53 1000 2050 0,33 0,00 0,00 0,00 0,33 2 gnome-shell 13:45:53 1000 3767 7,67 1,67 0,00 0,00 9,33 3 firefox 13:45:53 1000 3985 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co 13:45:53 1000 4277 0,67 0,00 0,00 0,00 0,67 7 WebExtensions 13:45:53 1000 18502 0,33 0,00 0,00 0,00 0,33 5 virt-manager 13:45:53 1000 560859 1,00 0,00 0,00 0,00 1,00 0 Isolated Web Co 13:45:53 1000 656930 0,33 0,00 0,00 0,00 0,33 5 Isolated Web Co 13:45:53 1000 1235407 0,33 0,00 0,00 0,00 0,33 6 Isolated Web Co 13:45:53 1000 1286455 0,67 0,67 0,00 0,00 1,33 7 Isolated Web Co 13:45:53 1000 1287953 0,00 0,33 0,00 0,00 0,33 1 pidstat ^C Average: UID PID %usr %system %guest %wait %CPU CPU Command Average: 0 638 0,00 0,67 0,00 0,00 0,67 - irq/204-nvidia Average: 0 945 0,00 0,11 0,00 0,00 0,11 - libvirtd Average: 1000 1744 0,11 1,00 0,00 0,00 1,11 - pulseaudio Average: 1000 2050 0,67 0,11 0,00 0,00 0,78 - gnome-shell Average: 1000 3767 5,77 1,44 0,00 0,00 7,21 - firefox Average: 1000 3985 0,55 0,22 0,00 0,00 0,78 - Isolated Web Co Average: 1000 3989 0,00 0,11 0,00 0,00 0,11 - Isolated Web Co Average: 1000 4277 0,55 0,00 0,00 0,00 0,55 - WebExtensions Average: 1000 18502 0,11 0,00 0,00 0,00 0,11 - virt-manager Average: 1000 25736 0,00 0,11 0,00 0,00 0,11 - Isolated Web Co Average: 1000 492449 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co Average: 1000 560859 0,55 0,00 0,00 0,00 0,55 - Isolated Web Co Average: 1000 656930 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co Average: 1000 657165 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co Average: 1000 858923 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co Average: 1000 1202040 0,11 0,00 0,00 0,00 0,11 - Isolated Web Co Average: 1000 1235407 0,33 0,00 0,00 0,00 0,33 - Isolated Web Co Average: 0 1284255 0,00 0,11 0,00 0,00 0,11 - kworker/7:1-events Average: 0 1284904 0,00 0,11 0,00 0,00 0,11 - kworker/0:2-pm Average: 0 1285798 0,00 0,11 0,00 0,00 0,11 - kworker/3:0-events Average: 1000 1286455 0,78 0,55 0,00 0,00 1,33 - Isolated Web Co Average: 0 1287603 0,00 0,11 0,00 0,00 0,11 - kworker/0:0-events Average: 1000 1287953 0,00 0,22 0,00 0,00 0,22 - pidstat Filter by process To zoom in on a particular process or task, the option -C or -G can be used. It filters on the provided string and looks if that is part of the command name. When needed, a regular expression can be used.\n# pidstat -C firefox 3 Linux 6.5.0-28-generic (ws03) 20-05-24 _x86_64_\t(8 CPU) 13:50:10 UID PID %usr %system %guest %wait %CPU CPU Command 13:50:13 1000 3767 7,97 1,99 0,00 0,00 9,97 2 firefox 13:50:13 UID PID %usr %system %guest %wait %CPU CPU Command 13:50:16 1000 3767 1,67 1,00 0,00 0,00 2,67 5 firefox 13:50:16 UID PID %usr %system %guest %wait %CPU CPU Command 13:50:19 1000 3767 3,33 1,00 0,00 0,00 4,33 2 firefox 13:50:19 UID PID %usr %system %guest %wait %CPU CPU Command 13:50:22 1000 3767 5,00 2,33 0,00 0,00 7,33 5 firefox ^C Average: UID PID %usr %system %guest %wait %CPU CPU Command Average: 1000 3767 4,50 1,58 0,00 0,00 6,08 - firefox To get more details, add the options -l for long format (command with arguments) or -t for task details. The last one will include a tree-based output.\n# pidstat -l -t 12:55:31 UID TGID TID %usr %system %guest %wait %CPU CPU Command 12:55:31 0 1 - 0.00 0.00 0.00 0.00 0.00 0 /sbin/init 12:55:31 0 - 1 0.00 0.00 0.00 0.00 0.00 0 |__systemd 12:55:31 0 2 - 0.00 0.00 0.00 0.00 0.00 0 kthreadd 12:55:31 0 - 2 0.00 0.00 0.00 0.00 0.00 0 |__kthreadd 12:55:31 0 13 - 0.00 0.00 0.00 0.00 0.00 0 ksoftirqd/0 12:55:31 0 - 13 0.00 0.00 0.00 0.00 0.00 0 |__ksoftirqd/0 12:55:31 0 14 - 0.00 0.00 0.00 0.00 0.00 0 rcu_sched 12:55:31 0 - 14 0.00 0.00 0.00 0.00 0.00 0 |__rcu_sched 12:55:31 0 15 - 0.00 0.00 0.00 0.00 0.00 0 migration/0 12:55:31 0 - 15 0.00 0.00 0.00 0.00 0.00 0 |__migration/0 12:55:31 0 22 - 0.00 0.00 0.00 0.00 0.00 0 khungtaskd 12:55:31 0 - 22 0.00 0.00 0.00 0.00 0.00 0 |__khungtaskd 12:55:31 0 25 - 0.00 0.00 0.00 0.00 0.00 0 kcompactd0 12:55:31 0 - 25 0.00 0.00 0.00 0.00 0.00 0 |__kcompactd0 12:55:31 0 83 - 0.00 0.00 0.00 0.00 0.00 0 kworker/0:1H-kblockd 12:55:31 0 - 83 0.00 0.00 0.00 0.00 0.00 0 |__kworker/0:1H-kblockd 12:55:31 0 294 - 0.00 0.00 0.00 0.00 0.00 0 jbd2/dm-0-8 12:55:31 0 - 294 0.00 0.00 0.00 0.00 0.00 0 |__jbd2/dm-0-8 12:55:31 0 369 - 0.00 0.00 0.00 0.00 0.00 0 /lib/systemd/systemd-journald 12:55:31 0 - 369 0.00 0.00 0.00 0.00 0.00 0 |__systemd-journal 12:55:31 0 406 - 0.00 0.00 0.00 0.00 0.01 0 /sbin/multipathd -d -s 12:55:31 0 - 406 0.00 0.00 0.00 0.00 0.00 0 |__multipathd 12:55:31 0 - 413 0.00 0.00 0.00 0.00 0.00 0 |__multipathd 12:55:31 0 - 414 0.00 0.00 0.00 0.00 0.00 0 |__multipathd Start and monitor a single process Sometimes you want to zoom in on a single process. With pidstat we can start the process and monitor it, until we manually stop it. This way the process can be monitored from beginning till the end.\n# pidstat 3 -e /opt/google/chrome/chrome Linux 6.5.0-28-generic (workstation) 20-05-24 _x86_64_\t(8 CPU) 14:20:29 UID PID %usr %system %guest %wait %CPU CPU Command 14:20:32 1000 1290789 42,00 6,00 0,00 2,00 48,00 7 chrome 14:20:35 1000 1290789 65,00 18,33 0,00 7,67 83,33 0 chrome 14:20:38 1000 1290789 1,00 0,67 0,00 0,00 1,67 7 chrome 14:20:41 1000 1290789 0,33 0,00 0,00 0,00 0,33 6 chrome 14:20:44 1000 1290789 1,67 0,33 0,00 0,00 2,00 5 chrome 14:20:47 1000 1290789 3,67 0,67 0,00 0,33 4,33 2 chrome 14:20:50 1000 1290789 8,00 4,00 0,00 0,33 12,00 2 chrome 14:20:53 1000 1290789 12,00 2,00 0,00 0,33 14,00 1 chrome 14:20:56 1000 1290789 5,00 1,67 0,00 0,00 6,67 2 chrome 14:20:59 1000 1290789 11,00 3,67 0,00 0,33 14,67 4 chrome 14:21:02 1000 1290789 9,67 3,33 0,00 0,33 13,00 2 chrome 14:21:05 1000 1290789 6,67 2,00 0,00 0,33 8,67 2 chrome Average: 1000 1290789 13,83 3,56 0,00 0,97 17,39 - chrome Got more useful commands to help troubleshooting performance issues? Contribute and make this article better.\n","permalink":"https://linux-audit.com/system-performance/cpu/","tags":["cpu","linux","performance"],"title":"Troubleshooting CPU usage"},{"categories":["Shell scripting"],"contents":"Within a shell script we can test for the presence of directories and files. In this article we look at the basics and the more exotic options available.\nTesting the presence and type of a file can be done using the test command. For shell scripts, it is more common to use the [ command. Yes, it is an actual command. However, it may also be available as a so-called builtin and part of the shell. To validate this for bash, run the compgen -b command.\nCheck if a file exists Let\u0026rsquo;s start with testing if a file exists. This can be done using the -e option.\nMYFILE=\u0026#34;/etc/passwd\u0026#34; if [ -e \u0026#34;${MYFILE}\u0026#34; ]; then echo \u0026#34;File exists\u0026#34; fi Very often we actually want to test if a file exists and is a regular file. In this common case it might be better to use the -f option instead.\nMYFILE=\u0026#34;/etc/passwd\u0026#34; if [ -f \u0026#34;${MYFILE}\u0026#34; ]; then echo \u0026#34;File exists an is a regular file\u0026#34; fi Is a file executable? MYFILE=\u0026#34;/etc/passwd\u0026#34; if [ -x \u0026#34;${MYFILE}\u0026#34; ]; then echo \u0026#34;File is executable\u0026#34; fi Check the presence of a directory ETCDIR=\u0026#34;/etc\u0026#34; if [ -d \u0026#34;${ETCDIR}\u0026#34; ]; then echo \u0026#34;Directory ${ETCDIR} exists\u0026#34; else echo \u0026#34;Error: Directory ${ETCDIR} does not exist\u0026#34; exit 1 fi Options Option Purpose -b File exists (type: block special) -c File exists (type: character special) -d File exists (type: directory) -e File exists -f File exists (regular file) -g File exists with setgid bit set -G File exists, owned by effective group ID -h File exists (type: symbolic link), similar to -L -k File exists, sticky bit set -L File exists (type: symbolic link), similar to -h -N File exists, modified since last read -O File exists, owned by effective user ID -p File exists (type: named pipe) -r File exists, readable -s File exists, size greater than 0 bytes -S File exists (type: socket) -t File descriptor is opened on a terminal, requires a file description instead of file -u File exists, setuid bit set -w File exists, write permission granted -x File exists, execute or search permission granted ","permalink":"https://linux-audit.com/shell-scripting/check-if-a-directory-or-file-exists/","tags":["linux","shell script"],"title":"Check if a directory or file exists"},{"categories":["Network","System Administration"],"contents":"When using systemd, it is common to have the resolve daemon being responsible for handling DNS requests. It might be possible that a previous query is cached in this local resolver. To clear out this cache, we need the resolvectl in combination with the flush-caches subcommand.\nClear DNS cache On modern systemd implementations, run the following command:\nresolvectl flush-caches\nUnder normal conditions, this command won\u0026rsquo;t show any result.\nIf your Linux distribution is older, then use systemd-resolve --flush-caches instead.\nTo confirm that the cache is empty, retrieve the statistics and look at the current cache size.\n# resolvectl statistics DNSSEC supported by current servers: no Transactions Current Transactions: 0 Total Transactions: 145185 Cache Current Cache Size: 0 Cache Hits: 30468 Cache Misses: 46205 DNSSEC Verdicts Secure: 0 Insecure: 0 Bogus: 0 Indeterminate: 0 In this case is the cache size back to zero, which is the expected result. The DNS cache is now empty and can be slowly filled again when new DNS requests are made.\n","permalink":"https://linux-audit.com/networking/faq/how-to-clear-dns-cache-with-systemd/","tags":["dns","faq","howto","network","resolvectl","systemd"],"title":"Show to clear the DNS cache with systemd"},{"categories":["System administration"],"contents":"","permalink":"https://linux-audit.com/system-administration/commands/resolvectl/","tags":["dns","network","networking","resolvectl","systemd"],"title":"resolvectl"},{"categories":["Network","System Administration"],"contents":"The nstat command can be used to retrieve information about network connection statistics and relevant counters.\nUsing the tool is easy, just run it without any parameters to see relevant counters.\n# nstat #kernel IpInReceives 865 0.0 IpInDelivers 864 0.0 IpOutRequests 1073 0.0 TcpInSegs 864 0.0 TcpOutSegs 1074 0.0 Ip6InReceives 1 0.0 Ip6InMcastPkts 1 0.0 Ip6InOctets 382 0.0 Ip6InMcastOctets 382 0.0 Ip6InNoECTPkts 1 0.0 TcpExtTCPHPHits 490 0.0 TcpExtTCPPureAcks 30 0.0 TcpExtTCPHPAcks 451 0.0 TcpExtTCPSACKReorder 1 0.0 TcpExtTCPBacklogCoalesce 48 0.0 TcpExtTCPSackShiftFallback 1 0.0 TcpExtTCPRcvCoalesce 190 0.0 TcpExtTCPOFOQueue 8 0.0 TcpExtTCPAutoCorking 106 0.0 TcpExtTCPOrigDataSent 909 0.0 TcpExtTCPKeepAlive 22 0.0 TcpExtTCPDelivered 910 0.0 TcpExtTCPAckCompressed 1 0.0 IpExtInOctets 205502 0.0 IpExtOutOctets 285260 0.0 IpExtInNoECTPkts 866 0.0 Now there is something special with this tool, that is maintains history. So each time you run the tool, it will show you the difference between the current and the previous run. Also, if you are just interesting in information about the TCP connections, then we need to fine-tune the output. One could use grep for this, but it is better to perform the specific query right away.\nShow TCP counters Relevant counters can be shown by providing the --reset option. This resets the history, so that when you run the command again, all counters are shown. Or in other words, the cumulative values of the counters.\nnstat --reset Tcp*\nShow all TCP counters To retrieve all counters, including those with a value of zero, add the --zeros option.\n# nstat --reset --zeros Tcp* #kernel TcpActiveOpens 339 0.0 TcpPassiveOpens 4218 0.0 TcpAttemptFails 32 0.0 TcpEstabResets 29 0.0 TcpInSegs 2583671 0.0 TcpOutSegs 3480725 0.0 TcpRetransSegs 325 0.0 TcpInErrs 0 0.0 TcpOutRsts 178703 0.0 TcpInCsumErrors 0 0.0 TcpExtSyncookiesSent 0 0.0 TcpExtSyncookiesRecv 0 0.0 TcpExtSyncookiesFailed 0 0.0 TcpExtEmbryonicRsts 8 0.0 TcpExtPruneCalled 0 0.0 TcpExtRcvPruned 0 0.0 TcpExtOfoPruned 0 0.0 TcpExtOutOfWindowIcmps 0 0.0 TcpExtLockDroppedIcmps 0 0.0 TcpExtArpFilter 0 0.0 TcpExtTW 2989 0.0 Got another method to retrieve relevant information? Let it know!\n","permalink":"https://linux-audit.com/networking/faq/how-to-show-all-tcp-network-statistics-and-counters/","tags":["faq","howto","network"],"title":"How to show network TCP statistics and counters"},{"categories":["System administration"],"contents":"The lscpu command retrieves information about the the CPU architecture. The sources to retrieve this information include /proc/cpuinfo , pseudo file system sysfs, and available libraries.\nExamples of information includes:\nCores CPUs BogoMIPS Byte order Cache sharing Caches Family Model Stepping NUMA nodes Threads Sockets Virtualization details Vulnerabilities The tool is both suitable for using in the terminal and manual interpretation, but also for automated processing. When its output is piped to other another application, it is aware that any screen markup should be left out.\nWith more vulnerabilities begin discovered in CPUs, the tool is also able to show applicable vulnerabilities discovered in the specific CPU architecture.\nExample output:\nArchitecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 40 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Vendor ID: GenuineIntel Model name: QEMU Virtual CPU version 2.5+ CPU family: 15 Model: 107 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 Stepping: 1 BogoMIPS: 6191.99 Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm cpuid_fault pti Virtualization features: Hypervisor vendor: KVM Virtualization type: full Caches (sum of all): L1d: 32 KiB (1 instance) L1i: 32 KiB (1 instance) L2: 4 MiB (1 instance) L3: 16 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: KVM: Mitigation: VMX unsupported L1tf: Mitigation; PTE Inversion Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Meltdown: Mitigation; PTI Mmio stale data: Unknown: No mitigations Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline Srbds: Not affected Tsx async abort: Not affected ","permalink":"https://linux-audit.com/system-administration/commands/lscpu/","tags":["cpu","hardware","linux"],"title":"lscpu"},{"categories":["Hardware","System Administration"],"contents":"Information about the CPU can be retrieved very easily on Linux systems. The aptly named lscpu lists details about the CPU.\nCPU details There is quite some information to pull in related to the CPU. It depends on the version of the tool, but information may include:\nArchitecture Vendor information Virtualization features Caches NUMA details CPU vulnerabilities # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Model name: Westmere E56xx/L56xx/X56xx (Nehalem-C) Stepping: 1 CPU MHz: 2394.454 BogoMIPS: 4788.90 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0,1 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm kaiser arat A newer version will group information and show vulnerabilities that may be applicable to the CPU.\n# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 40 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Vendor ID: GenuineIntel Model name: QEMU Virtual CPU version 2.5+ CPU family: 15 Model: 107 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 Stepping: 1 BogoMIPS: 6191.99 Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc nopl xtopology cpuid tsc_known_freq pni s sse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm cpuid_fault pti Virtualization features: Hypervisor vendor: KVM Virtualization type: full Caches (sum of all): L1d: 32 KiB (1 instance) L1i: 32 KiB (1 instance) L2: 4 MiB (1 instance) L3: 16 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: KVM: Mitigation: VMX unsupported L1tf: Mitigation; PTE Inversion Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Meltdown: Mitigation; PTI Mmio stale data: Unknown: No mitigations Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline Srbds: Not affected Tsx async abort: Not affected As can be seen from this output, the lscpu provides a lot of details even without any options specified. The level of details increases when using newer versions.\n","permalink":"https://linux-audit.com/hardware/faq/how-to-see-cpu-details/","tags":["cpu","faq","hardware","howto"],"title":"How to see CPU details"},{"categories":["System administration"],"contents":"Users of the command-line are typically aware of the netstat command. The package that netstat belongs to is getting deprecated and slowly replaced with tools like ip. This is also where nstat comes in, a more modern approach to pulling in data from the kernel. Another benefit is that it can retrieve more information than netstat. This is due to the static list of metrics that netstat looks for, while nstat will parse output files from /proc.\nSources that nstat uses to retrieve its information include the following files.\n/proc/net/netstat /proc/net/snmp /proc/net/snmp6 /proc/net/sctp/snmp /proc/uptime ","permalink":"https://linux-audit.com/system-administration/commands/nstat/","tags":["linux","networking","statistics","troubleshooting","uptime"],"title":"nstat"},{"categories":["System administration"],"contents":"","permalink":"https://linux-audit.com/system-administration/commands/vmstat/","tags":["memory"],"title":"vmstat: system statistics about memory, processes, disks, block IO, CPU, and more"},{"categories":null,"contents":"Physical RAM is used to store information. Linux divides this RAM into smaller chunks, named memory pages. When there is no more normal memory available, the Linux kernel might need to temporarily store information aside. This is called paging or swap space.\nDuring the process of paging, memory pages will be moved from the RAM to the disk. This way memory is freed up for active processes, while older information is temporarily stored on the disk. A disk is much slower than RAM, so typically you want to avoid paging as much as possible.\nSwap size and usage Basic information about the size of the swap is available via free. It also a quick overview of memory details. The source of the information is the file /proc/meminfo .\n# free total used free shared buff/cache available Mem: 980412 169752 211584 988 599076 661372 Swap: 1959932 42216 1917716 Slightly more detailed information can be retrieved using the vmstat command.\n# vmstat --stats | grep swap 529192 K swap cache 1959932 K total swap 42216 K used swap 1917716 K free swap 18986 pages swapped in 32724 pages swapped out Which processes are using the most swap? To see which processes are using swap, consider using smem to retrieve the details.\nsmem --columns=\u0026quot;pid swap\u0026quot;\nAn alternative is to do it without, and select a process that has high memory usage (e.g. from top output). Retrieve the pid with pidof or pgrep, then look in the \u0026lsquo;/proc/PID/status\u0026rsquo; file.\nTo automate this step, use a script like this:\nfor I in $(pidof firefox); do awk -v PID=${I} '/^VmSwap/{print PID\u0026quot;=\u0026quot;$2$3}' /proc/${I}/status; done\nWhen in doubt what processes are using swap, we can retrieve the output for all processes.\nawk \u0026#39;/Name|VmSwap/{printf (/Name/?\u0026#34;\\n\u0026#34;:\u0026#34; \u0026#34;)$2$3} END{ print \u0026#34;\\n\u0026#34;}\u0026#39; /proc/*/status \\ | awk \u0026#39;NF\u0026gt;1\u0026#39; \\ | sort --human-numeric-sort --key=2 --reverse \\ | column --table This command retrieves the lines with Name or VmSwap and replaces the line feeds, so that we can print the name, and the size on one line. Some kernel processes will not have a VmSwap value, so those entries we filter out by only showing the entries that have two fields (name and swap size). Finally we sort on the second column, and present a nice table.\n","permalink":"https://linux-audit.com/system-performance/memory/swap/","tags":["awk","howto","linux","memory","performance","ram","swap","system performance"],"title":"Swap memory information"},{"categories":["Kernel"],"contents":"This sysctl key controls the use of the performance events system. It restricts what actions an unprivileged user can do when using the perf tools. This setting is useful to limit the access to possibly sensitive information that can be gathered from the kernel and processes.\nDebian-based systems may have higher numbers available than the current upper limit of 2, which is also the default value.\n","permalink":"https://linux-audit.com/kernel/sysctl/kernel/kernel.perf_event_paranoid/","tags":["kernel","kernel hardening","linux","sysctl"],"title":"Sysctl: kernel.perf_event_paranoid"},{"categories":["Network","System Administration"],"contents":"To see all network traffic happening on a Linux system or the network, the tcpdump tool is a seasoned tool for the task. For network engineers it might be easy to use, but for the average person the amount of options might be overwhelming. This cheat sheet dives into what tcpdump can do, with examples that are often used to troubleshoot issues and monitor the network.\nBasic options Some of the common options to use on tcpdump include:\nShort option Long option What the option does -i IFACE --interface=IFACE Select IFACE as the interface on which to capture -D --list-interfaces Show available interfaces that can be used -n Do not resolve hostnames, protocols, etc. -q Quick/quiet output -r FILE.pcap Read an earlier packet capture session -v Verbose output -vv More verbose -vvv Most verbose -w FILE.pcap Store the captured packets in a file Creating a shell script? Then we suggest using the long format option, as this improves the readability. For quick use of on the command-line consider using the short notation of the related option.\nFilter expressions By only providing options, tcpdump will display all relevant captured packets. This is typically not what we want, especially when we want to zoom in on a specific host or protocol. For this purpose tcpdump uses filter expressions.\nFilter Intended goal Example host Filter by source or destination host tcpdump host 10.0.1.1 port Filter by port number or service name tcpdump port 80 src host Filter by source host tcpdump src host 10.2.3.1 Operators In the filter expressions it is common to combine multiple filters. For example, one of the communicating hosts in combination with a port. Some of the operators include:\nnot (!) and (\u0026amp;\u0026amp;) or (||) less (\u0026lt;) greater (\u0026gt;) If two filters need to match, we can use with an AND operator:\ntcpdump host 10.2.3.1 and port 80\nWhen we throw in a \u0026rsquo;not\u0026rsquo;, we can see all traffic for one host, except SSH traffic using a combination:\ntcpdump host 10.4.2.3 and not port 22\nProtocols Some protocols can be defined as-is, limiting output to only those protocols.\nicmp tcp udp Interfaces and traffic isolation Typically a system has more than one interface available. To prevent being overloaded with too much traffic, it is wise to isolate the traffic. This can be done with the --interface= by defining the specific interface on which network packets is captured.\nShow available interfaces Modern Linux distributions have typically more interfaces than you would initially think. Use the option --list-interfaces to display all available interfaces, including those for netfilter, Bluetooth, D-Bus.\n# tcpdump --list-interfaces 1.ens3 [Up, Running, Connected] 2.any (Pseudo-device that captures on all interfaces) [Up, Running] 3.lo [Up, Running, Loopback] 4.bluetooth-monitor (Bluetooth Linux Monitor) [Wireless] 5.nflog (Linux netfilter log (NFLOG) interface) [none] 6.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none] 7.dbus-system (D-Bus system bus) [none] 8.dbus-session (D-Bus session bus) [none] Capture on all interfaces To capture traffic on all interface, use \u0026lsquo;any\u0026rsquo; as value.\ntcpdump -n -i any\nReducing output For troubleshooting and a first analysis, it may be useful to limit the amount of output. With option -c a count can be set and reduced the number of captured packets.\ntcpdump -n -c 10\nIncreasing output details tcpdump -nvv port 80\nFilter packets By host When zooming in on a particular system, we can use host and specify the related host.\ntcpdump -n host 192.168.178.16\nOnly looking for packets coming from a specific host? Add the src statement as well.\ntcpdump -n src host 192.168.1.19\nBy port Define the port by its protocol name or number. For HTTP connections we could use:\ntcpdump -n port http\nOr its alternative, by port number:\ntcpdump -n port 80\nBy protocol Looking for all TCP connections to or from a system? Set the filter to tcp and all other protocols will be ignored.\ntcpdump -n tcp\nTCP flags New connections using TCP have multiple flags available, each depending on the state of the connection. Newly created connections have the SYN flag active, so are a great way to filter out all new connections.\ntcpdump -i ens18 -n 'tcp[tcpflags] == tcp-syn'\nWe can also filter specifically the SYN/ACK state, which happens at the beginning.\ntcpdump -i ens18 'tcp[13] = 18'\nThe value 13 comes from the TCP header. In byte 13 the TCP flags are stored.\nUseful values to know\nURG = 32 ACK = 16 PSH = 8 RST = 4 SYN = 2 FIN = 1 So seeing all TCP connections that come to an end (finish):\n# tcpdump \u0026#39;tcp[13] \u0026amp; 1 != 0\u0026#39; 09:30:53.270658 IP 192.168.1.11.33392 \u0026gt; 192.168.1.12.22: Flags [F.], seq 2535494855, ack 1906495977, win 501, options [nop,nop,TS val 3548246170 ecr 1901198083], length 0 ARP tcpdump -n ether proto 0x0806\nor easier:\ntcpdump -n arp\nIPv6 tcpdump ip6\nPacket size We can also filter on size, which is in the case of tcpdump the total length. This includes link layer, IP, and for example TCP headers.\ntcpdump -n len greater 1000\nDepending on the protocol, you have to carefully look (and monitor) what intended length you are searching for. See UDP example below.\nUDP UDP does not have a length specified, so if you are looking to filter for those specifically, a little bit of size counting is needed.\nIP header is at a minimum 20 bytes and 60 bytes maximum UDP header is 8 bytes UDP payload So to filter for very small UDP packets of 4 bytes or less (20+8+4=32), we can use:\ntcpdump -n 'ip[2:2] \u0026lt;= 32 and udp'\nCombining filters See all traffic of a particular host, but ignore the SSH connection.\ntcpdump -n host 192.168.178.16 and port not 22\nUsing files Using filters from a file When doing repeating captures, the option -F helps to get a filter expression from an external file. This is very useful for constructing more specific filters that are harder to remember.\ntcpdump -n -c 10 -F tcpdump-filter-arp-only\nStore output in a file Tcpdump allows to store a capture in a PCAP file. This file format can be used with other programs, like Wireshark .\ntcpdump -n -c 10 -w for-later-analysis.pcap\nTo read the packet capture, tcpdump can also be used again.\ntcpdump -r for-later-analysis.pcap\n","permalink":"https://linux-audit.com/cheat-sheets/tcpdump/","tags":["arp","cheatsheet","howto","ipv6","networking","one-liner","tcpdump"],"title":"tcpdump cheat sheet"},{"categories":["System Administration"],"contents":"When connecting with a system the first time, you may not know what Linux distribution or version is running. Another possibility is that you know the Linux distribution, but not exactly what version is running. With a few handy tools, that is fairly easy to discover.\nHostnamectl Modern Linux distributions are typically using systemd as its service and system manager. In that case, the hostnamectl command will be of great help.\n# hostnamectl Static hostname: mysystem Icon name: computer-vm Chassis: vm Machine ID: 4e1243bd22c66e76c2ba9eddc1f91394 Boot ID: 9054fbe0b622c638224d50d20824d2ff Virtualization: kvm Operating System: Ubuntu 22.04.4 LTS Kernel: Linux 5.15.0-106-generic Architecture: x86-64 Hardware Vendor: QEMU Hardware Model: Standard PC _i440FX + PIIX, 1996_ This virtual system is using Ubuntu 22.04 LTS.\nWant to test it on a remote system?\nhostnamectl --host=myothersystem\nlsb_release or lsb-release file No luck with hostnamectl? Then another option might be the hostnamectl with the --all option.\n# lsb_release --all No LSB modules are available. Distributor ID:\tUbuntu Description:\tUbuntu 22.04.4 LTS Release:\t22.04 Codename:\tjammy In this case, we get a similar result.\nUsing a release or version file An alternative to the command above, is that on some systems the file /etc/lsb-release exists. If not, then check if a similarly named file exist with the Linux distribution name in it.\nls -l /etc/*-release\nIf you get a result, cat the related file and it should typically show the Linux distribution and version details.\nOutput of uname Another option that might give away a hint, is by running the uname command.\n# uname -a Linux mysystem 6.1.0-20-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux The output reveals it is a Debian system, but not the actual version. So this option gives at least a start, but requires additional research.\n","permalink":"https://linux-audit.com/system-administration/methods-to-find-the-linux-distribution-and-version/","tags":["faq","hostname","hostnamectl","howto","linux"],"title":"Methods to find the Linux distribution and version"},{"categories":["Hardware","System Administration"],"contents":"The dmidecode command is a DMI table decoder and makes the available information human readable. The related specification SMBIOS defines the available DMI types.\nBasics Dmidecode can query a several pieces of information. One of the methods to query is using the hardware type. This uses the --type followed by a number or keyword.\nWhen using dmidecode, root privileges are usually required to read the information.\nTypes and keywords Type Keyword Information 0 bios BIOS 1 system System 2 baseboard Baseboard 3 chassis Chassis 4 processor Processor 5 memory Memory Controller 6 memory Memory Module 7 cache Cache 8 connector Port Connector 9 slot System Slots 10 baseboard On Board Devices 11 - OEM Strings 12 system System Configuration Options 13 bios BIOS Language 14 - Group Associations 15 system System Event Log 16 memory Physical Memory Array 17 memory Memory Device 18 - 32-bit Memory Error 19 - Memory Array Mapped Address 20 - Memory Device Mapped Address 21 - Built-in Pointing Device 22 - Portable Battery 23 system System Reset 24 - Hardware Security 25 - System Power Controls 26 - Voltage Probe 27 - Cooling Device 28 - Temperature Probe 29 - Electrical Current Probe 30 - Out-of-band Remote Access 31 - Boot Integrity Services 32 system System Boot 33 - 64-bit Memory Error 34 - Management Device 35 - Management Device Component 36 - Management Device Threshold Data 37 - Memory Channel 38 - IPMI Device 39 - Power Supply 40 - Additional Information 41 baseboard Onboard Devices Extended Information 42 - Management Controller Host Interface Strings Besides the types, it is also possible to provide a keyword to the --string option. This will reveal very specific details about the system, typically a single item.\nbaseboard-asset-tag baseboard-manufacturer baseboard-product-name baseboard-serial-number baseboard-version bios-release-date bios-revision bios-vendor bios-version chassis-asset-tag chassis-manufacturer chassis-serial-number chassis-type chassis-version firmware-revision processor-family processor-frequency processor-manufacturer processor-version system-family system-manufacturer system-product-name system-version system-serial-number system-sku-number system-uuid BIOS The information about the BIOS can be displayed with --typebios, which includes several types.\n# dmidecode --type bios # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Handle 0x0000, DMI type 0, 26 bytes BIOS Information Vendor: Intel Corp. Version: PXXXXX.0000.111.1234 Release Date: 01/24/2024 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 0 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported Japanese floppy for NEC 9800 1.2 MB is supported (int 13h) Japanese floppy for Toshiba 1.2 MB is supported (int 13h) 5.25\u0026#34;/360 kB floppy services are supported (int 13h) 5.25\u0026#34;/1.2 MB floppy services are supported (int 13h) 3.5\u0026#34;/720 kB floppy services are supported (int 13h) 3.5\u0026#34;/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 7.23 Firmware Revision: 12.23 Handle 0x0031, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Long Installable Languages: 1 en|US|iso8859-1 Currently Installed Language: en|US|iso8859-1 Handle 0x0058, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS BIOS version The version of the BIOS can be retrieved using --stringbios-version and will only return its value. Great for automation purposes.\ndmidecode --string bios-version\nMemory Show all memory information can be done using the memory keyword.\ndmidecode --type memory\nMemory modules To list only the memory modules, define the numeric type.\ndmidecode --type 17\n","permalink":"https://linux-audit.com/cheat-sheets/dmidecode/","tags":["bios","cheatsheet","cpu","hardware","howto","linux","memory","one-liner"],"title":"dmidecode cheat sheet"},{"categories":["Hardware","System Administration"],"contents":"Finding out the memory specifications on a Linux system is easy when using the dmidecode command. Let\u0026rsquo;s have a look at what information can be extracted and how to zoom in on some specifics, like memory type and memory speed.\nShow all memory The first option to use is --typememory to retrieve information about the memory, but also from the motherboard.\n# dmidecode --type memory # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Handle 0x0042, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: None Maximum Capacity: 64 GB Error Information Handle: Not Provided Number Of Devices: 2 snipped The output will show all information related to memory, including the relevant parts of the motherboard, including the maximum capacity of memory.\nThe keyword memory includes:\nType 5: Memory Controller Type 6: Memory Module Type 16: Physical Memory Array Type 17: Memory Device Show the memory modules If you are interested in the memory modules, then it might be better to specify --type17, or the memory devices.\n# dmidecode --type 17 # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Handle 0x0047, DMI type 17, 92 bytes Memory Device Array Handle: 0x0042 Error Information Handle: Not Provided Total Width: 64 bits Data Width: 64 bits Size: 32 GB Form Factor: SODIMM Set: None Locator: Controller0-ChannelA-DIMM0 Bank Locator: BANK 0 Type: DDR4 Type Detail: Synchronous Speed: 3200 MT/s Manufacturer: Micron Technology Serial Number: ABCDABCD Asset Tag: 9876543210 Part Number: 16ATF4G64HZ-3G2F1 Rank: 2 Configured Memory Speed: 3200 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Not Specified Module Manufacturer ID: Bank 1, Hex 0x2C Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 32767 MB Cache Size: None Logical Size: None Show only type and speed To quickly extract some details such type and speed, we can filter out these rows from the output.\n# dmidecode --type 17 | grep -E \u0026#39;Type|Speed\u0026#39; Type: DDR4 Type Detail: Synchronous Speed: 3200 MT/s Configured Memory Speed: 3200 MT/s Type: DDR4 Type Detail: Synchronous Speed: 3200 MT/s Configured Memory Speed: 3200 MT/s ","permalink":"https://linux-audit.com/hardware/faq/how-to-see-memory-information-such-as-type-and-speed/","tags":["bios","faq","hardware","howto","linux","ram"],"title":"How to see memory information such as type and speed"},{"categories":["File systems","Linux forensics","System Administration"],"contents":"Let\u0026rsquo;s say you have a file with sensitive data. You want to delete it, but also be sure that it can\u0026rsquo;t be retrieve again. Instead of just removing the file with the rm command, there is a better option: shred\nIntroduction into shred Shred is a tool meant to clear the contents of a file. Instead of replacing it with zeroes, it uses random data. It does this, by default, with 3 passes. This should normally be enough to really purge any remaining bit of the original data. Optionally, shred can also delete the file.\nKnowing about this tool can be helpful when removing sensitive data of your customers, or deleting old data. It could also be an action before doing a full disk wipe, before bringing it to a recycle station.\nShred example So, how to use shred? Let\u0026rsquo;s have a look in how easy it is to use the tool. Before you do, only practice on a dummy file first.\nCreate a test file The first step is to create a test file.\necho test \u0026gt; test\nLet\u0026rsquo;s have a look what type of data is in our newly created file using the file command.\n# file test test: ASCII text Nothing exciting yet. Let\u0026rsquo;s look at some file statistics:\n# stat test File: test Size: 5 Blocks: 8 IO Block: 4096 regular file Device: 10303h/66307d\tInode: 22945816 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ michael) Gid: ( 1000/ michael) Access: 2024-05-09 19:39:49.903827717 +0200 Modify: 2024-05-09 19:39:49.903827717 +0200 Change: 2024-05-09 19:39:49.903827717 +0200 Birth: 2024-05-09 19:39:49.903827717 +0200 So our file is 5 bytes in length, just ordinary text, followed by a new line (\\n).\nShred the file Now we use shred to purge the content of the file by overwriting it with random data.\nshred test\nIs it still normal text?\n# file test test: data The answer is obvious: due to the random data, it is no longer a piece of ASCII text. Let\u0026rsquo;s have a look at the file statistics of our altered file.\n# stat test File: test Size: 4096 Blocks: 8 IO Block: 4096 regular file Device: 10303h/66307d\tInode: 22945816 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ michael) Gid: ( 1000/ michael) Access: 2024-05-09 19:39:49.903827717 +0200 Modify: 2024-05-09 19:39:54.935868836 +0200 Change: 2024-05-09 19:39:54.935868836 +0200 Birth: 2024-05-09 19:39:49.903827717 +0200 So not only has the contents been changed, it also grew in size. It exactly filled up a full IO block of 4096 bytes.\nAnother interesting fact is that the data from the file was not even read, as the Access timestamp shows the same information as before.\nBy using the command, the data and meta-data changed. So Modify and Change will reflect these changes by updating the related timestamps.\nShred and delete Want to shred the information and also delete it?\nshred --remove test\nThe file is gone.\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-securely-delete-a-file-and-its-contents/","tags":["faq","file system","forensics","howto","linux","privacy"],"title":"How to securely delete a file and its contents"},{"categories":["File systems","Linux forensics","System Administration"],"contents":"The creation date of a file can be displayed on a Linux system using the stat command. That is, if the file system on the Linux system supports this particular piece of information. If so, then it is stored in the btime or birth time. In 2017 this initial support was added.\nShowing the birth time of a file One option is to run stat with only the file name as its parameter.\n# stat /etc/passwd File: /etc/passwd Size: 1993 Blocks: 8 IO Block: 4096 regular file Device: fd00h/64768d\tInode: 132674 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-08 19:17:01.959352190 +0000 Modify: 2024-02-02 19:14:45.316650111 +0000 Change: 2024-02-02 19:14:45.320650046 +0000 Birth: 2024-02-02 19:14:45.316650111 +0000 The output will display the time stamps. If the birth time is available, it will be displayed on the line starting with Birth:, followed by a human readable value. If the birth time is not available, you may see a dash, empty value, or a zero (0).\nIf you just want to see the related line, use the --format option.\n# stat --format=%w /etc/passwd 2024-02-02 19:14:45.316650111 +0000 Prefer a Unix timestamp (time in seconds since epoch) instead?\n# stat --format=%W /etc/passwd 1706901285 ","permalink":"https://linux-audit.com/filesystems/faq/how-to-see-the-creation-date-of-a-file/","tags":["faq","file system","forensics","howto","linux"],"title":"How to see the creation date of a file"},{"categories":["File Systems","System Administration"],"contents":"The stat command can be used to show file statistics, such as file size, ownership, type, and several timestamps. It is a great addition to ls. Time to let it work for us!\nBasic example If we use the command on our /etc/passwd file, we might get output like below.\n# stat /etc/passwd File: /etc/passwd Size: 3387 Blocks: 8 IO Block: 4096 regular file Device: 10303h/66307d\tInode: 47186412 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-08 15:17:01.121123879 +0200 Modify: 2024-03-04 14:01:44.569913427 +0100 Change: 2024-03-04 14:01:44.577913496 +0100 Birth: 2024-03-04 14:01:44.569913427 +0100 Explanation of the fields Let\u0026rsquo;s have a look at all the fields and understand what the shared information means.\nFile The field File is the file name. Nothing really exciting here.\nSize The field Size represents the file size. This value is in bytes and similar to the output of a command like ls.\nBlocks The field Blocks is the first one where things get more interesting. It represents the number of reserved blocks to store data and meta-data about the file. It is related to the field IO Block that defines the size of a block to read or write from disk. The big difference is that a block is 512 bytes (sector size), where the IO block is usually 4096 bytes. In other words, those 8 blocks make up the 4096 bytes (4KiB) in total.\nWhen an empty file is created, the number of blocks is zero. When just a little bit of data is stored, you would normally expect the block to increase to one. But as the file system uses IO blocks of 4096 bytes, it needs 8 blocks.\nType Behind the IO Block, our example states regular file. One might expect this output after the field File, but instead it listed at the second line. The value could be one of these types:\nblock special file character special file directory fifo port regular file regular empty file semaphore shared memory object socket symbolic link timed memory object Device The Devicee field is the device displayed. The format is hexadecimal and refers to the device on which the file resides.\nInode Not really surprising, but the field Inode refers to inode within the file system. The value is normally unique, unless there are multiple hard links to the same inode. Or, in other words, if there are more file names that point to the exact same data.\nLinks The field Links is related to the field above. It shows how many files point to this inode. If it is showing 1, then there is only one file name pointing to this inode.\nAccess With the field Access we get a familiar output of the ls command. Both in an octal and human readable format.\nUid and Gid The field Uid refers to the owner (User ID), and Gid to the group (Group ID). Both displayed as a number and the user or group name.\nTime stamps Access time Another line that starts with \u0026lsquo;Access:\u0026rsquo;, but this time referring to the access time. If you read the content of a file, then this attribute will be updated.\nModify time The modify time refers to the time the actual contents of the file was changed.\nChange time Often the change and modification time are the same or very close. The change time is updated when meta-data of the file is updated. When we alter the content of a file, the access time, modification time, and change time are updated.\nBirth time As the name implies, the birth time is the moment of when a file was initially created. This value should normally only be set once.\nTesting stat on a practice file Let\u0026rsquo;s create an empty file and check the file information:\n# touch test # stat test File: test Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 10303h/66307d\tInode: 22968322 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ michael) Gid: ( 1000/ michael) Access: 2024-05-08 20:35:31.526876522 +0200 Modify: 2024-05-08 20:35:31.526876522 +0200 Change: 2024-05-08 20:35:31.526876522 +0200 Birth: 2024-05-08 20:35:31.526876522 +0200 We can see in the output that we have a \u0026lsquo;regular empty file\u0026rsquo;. The file size is zero, but so is the number of blocks! That is surprising, as the creation of a file needs to be stored somewhere, right? In this case the information is stored in the block of the inode. As there is no data in the file, no IO blocks had to be claimed. All timestamps are the same, which is to be expected as we created the file and performed no other actions with it.\nLet\u0026rsquo;s add a bit of text to the file:\n# echo \u0026#34;test\u0026#34; \u0026gt; test # stat test File: test Size: 5 Blocks: 8 IO Block: 4096 regular file Device: 10303h/66307d\tInode: 22968322 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ michael) Gid: ( 1000/ michael) Access: 2024-05-08 20:35:31.526876522 +0200 Modify: 2024-05-08 20:35:52.199129382 +0200 Change: 2024-05-08 20:35:52.199129382 +0200 Birth: 2024-05-08 20:35:31.526876522 +0200 Now the number of blocks went to 8, as expected. The file size is no 4, but 5 bytes. Wondering why? Run \u0026lsquo;hexdump -c\u0026rsquo; on the file to find the answer.\n","permalink":"https://linux-audit.com/filesystems/understanding-the-output-of-the-stat-command/","tags":["file permissions","file system","inode"],"title":"Understanding the output of the stat command"},{"categories":["System Administration"],"contents":"The tar command is common on Linux and UNIX based systems. It dates to Version 7 of Unix, and had its introduction in January 1979. Tar allows the system administrator to create file archives, similar to a ZIP file. In this cheat sheet we look at how to use the command, including several useful examples.\nBasics The tar command can be used by just a few options, followed by an archive name, optionally followed by another set of options.\nOptions Option What the option does -c Create archive -f Define file name of archive -J Compress using xz (may need newer version of tar) -v Verbose output -z Compress using gzip --zstd Compress using zstd (may need newer version of tar) A few remarks when using tar and the listed examples:\nThe options can be typically used with hyphens or without. Tar on Linux has other options than tar on systems like BSD, so consider this when an archive may be used on other systems. Typically compression is saving storage space, but it may not be needed when archiving already compressed files. In these examples gzip compression is used. Newer tar versions have also the option to compress using more modern compression methods that save more space. Tips when working with tar Tar has the option -v to display more verbose output. This may be helpful, but can come at the cost of speed. When using big archives, consider to leave out this option for additional performance.\nThe order of options is very flexible, so it is possible to put options before or after the archive file name. Important is that the -f option is in front of the file name.\nCreating archive To create an archive using tar, there are multiple options. The first one is that we can go to the directory that we want to archive, and put the archive itself one level higher.\ncd my-directory \u0026amp;\u0026amp; tar czf ../archive.tar.gz .\nAnother option is to define the working directory with -C which is then considered as the root of the directory structure.\ntar czf /root/mybackup-of-etc-systemd.tar.gz -C /etc systemd\nSlightly similar, but with storing the full path to the /etc/systemd directory:\ntar czf /root/mybackup-of-etc-systemd.tar.gz /etc/systemd\nShow contents of archive Before taking an action on an archive made by tar, it is wise to inspect it first. Typically the file extension tells you if it is compressed or not. Secondly, we can see the contents of an archive, so that we know the full paths of the files included. This is important when extracting the file, so the directory structure will be in the right place.\ntar tzf archive.tar.gz\nExtracting archive To extract the archive in the current directory.\ntar xzf archive.tar.gz\nDefine location where to extract Sometimes the contents of the archive should be extracted into a different location where archive resides. In that case, go to the (new) directory where you want to extract. Define where the archive is and that the extraction should place in the current directory. This is helpful when the archive is big or located on another file system.\ncd /path/to/extract \u0026amp;\u0026amp; tar xzf /path/to/archive.tar.gz -C .\nAnother option is define the location while the current work directory is the same as where the archive is. Short and effective.\ntar xzf archive.tar.gz -C /path/to/extract\nDepending on how and where the file will be extracted, it may be beneficial to have the full path or just the directory name.\nSingle file Extract a single file or directory from the archive\ntar xzf archive.tar.gz \u0026ldquo;file1\u0026rdquo;\nUsing wildcard To retrieve only a specific file type from the archive, a wildcard can be used.\ntar tzf archive.tar.gz --wildcards '*.txt'\nDo you have other good one-liners that everyone should know?\n","permalink":"https://linux-audit.com/cheat-sheets/tar/","tags":["cheatsheet","howto","one-liner"],"title":"tar cheat sheet"},{"categories":["Kernel","System Administration"],"contents":"Don\u0026rsquo;t know what a tainted kernel is? Have a look at the article explaining what a tainted kernel is first.\nWhat is causing the tainted kernel? If you have a tainted kernel, it will typically show up in the output of dmesg and starts with \u0026lsquo;Tainted:\u0026rsquo; followed by some letter(s). Use the lookup table below to find the related cause.\nBit Letter Number value Reason 0 G/P 1 Proprietary kernel module loaded 1 F 2 Kernel module was force loaded 2 S 4 SMP kernel oops on officially SMP incapable processor 3 R 8 Kernel module force unloaded 4 M 16 Processor reported a Machine Check Exception 5 B 32 Bad page referenced or some unexpected page flags 6 U 64 Taint requested by user space application 7 D 128 Kernel died recently (OOPS, bug) 8 A 256 ACPI table overridden by user 9 W 512 Kernel issued warning 10 C 1024 Staging driver loaded 11 I 2048 Workaround for bug in platform firmware applied 12 O 4096 Externally-built kernel module loaded 13 E 8192 Unsigned module loaded 14 L 16384 Soft lockup occurred 15 K 32768 Kernel live patched 16 X 65536 Auxiliary taint, defined for and used by Linux distributions 17 T 131072 Kernel was built with the struct randomization plugin Source: kernel.org\nAnother option is to check the value of /proc/sys/kernel/tainted. This numeric value is easy to lookup in the table if there is just a single cause. When there are multiple causes, like a live patched kernel already experienced a serious issue, then it may be harder to find. In that case, run the following script.\nfor i in $(seq 18); do echo $(($i-1)) $(($(cat /proc/sys/kernel/tainted)\u0026gt;\u0026gt;($i-1)\u0026amp;1));done\nThis for-loop will show each bit. The bits with the value of \u0026lsquo;1\u0026rsquo; then can be looked up. If bit 1 and 15 show up, it was a live patch (15) and also kernel module that was loaded (forced).\n","permalink":"https://linux-audit.com/kernel/faq/how-to-find-the-specific-cause-of-a-tainted-kernel/","tags":["faq","linux","kernel"],"title":"How to find the specific cause of a tainted kernel"},{"categories":["Kernel","System Administration"],"contents":"Linux uses the concept of a tainted kernel when specific events occurred. The word tainted means it is contaminated or polluted. Not in an environmental way, but in the sense that proper troubleshooting is no longer possible. Or not as reliable as one would like it to be.\nRunning a tainted kernel is not a problem usually. It is just a marker that something happened on the system itself. As long as a system is running stable, then one could accept continuing running in this state.\nCauses There are multiple causes why the kernel was marked as tainted. For example when live patching is used, parts of the kernel are swapped out or redirected. As this would seriously complicate debugging, the kernel marks itself tainted to indicate that reliable troubleshooting is not possible.\nResolving the issue The easiest and only way to clear the tainted state is a simple reboot of the system.\nBefore you do the reboot, you may want to know the specific reason why the kernel was tainted in the first place.\n","permalink":"https://linux-audit.com/kernel/faq/what-is-a-tainted-kernel/","tags":["faq","linux","kernel"],"title":"What is a tainted kernel"},{"categories":["Kernel"],"contents":"This is a setting that you would normally not change. It is a state that shows if the kernel was tainted. If the value is higher than zero, than it is tainted.\nRelevant FAQ: What is a tainted kernel?\n","permalink":"https://linux-audit.com/kernel/sysctl/kernel/kernel.tainted/","tags":["linux","kernel","sysctl"],"title":"kernel.tainted"},{"categories":["Network","System Administration"],"contents":"The purpose of the ip command on Linux is to show and alter network devices, interfaces, network routing, and tunnels. It can be used as a replacement of tools like arp, netstat, and route. As there is so much possible with this command, this cheat sheet tries to collect them for easier reference.\nBasics The main ip command uses subcommands and options. The last one is usually optional, unless you more information or details is needed.\nSome of the primary subcommands include:\naddress - IP protocol information (replacement for ifconfig command) link - Network device information neighbour - ARP and NDISC information (replacement for arp command) route - Routing table information (replacement for route command) When using ip, you can use full names or abbreviated ones. In this cheat sheet the full names will be listed first and later replaced by their shorter versions. For example, ip link, ip li and ip l will all give the same output.\nSubcommands Subcommands define a particular area within networking, such as the physical link, addressing, or routing.\nCommand Short version Goal Replaces ip address ip a Show IP address details ifconfig ip link ip l Show network link details (MAC) ifconfig ip maddress ip m Show multicast details netstat -g ip neighbour ip n Show other systems on network segment (ARP) arp ip route ip r Display routing information netstat -r or route ip tcp_metrics ip tc Display TCP caching information ? Options Long option Short option What the option does -details -d Show more detailed output, usually insightful for troubleshooting purposes. -Numeric -N Numeric output, no conversion of names (e.g. ports) -statistics -s Show statistics. The long format option can be abbreviated also by -stats. Creating a shell script? Then we suggest using the long format option, as this improves the readability. For quick use of on the command-line consider using the short notation of the related option.\nNetwork devices Show the available network devices with subcommand link. It includes information like the name of the network interface, optional alias, MAC address, MTU size, and its state (up/down).\n# ip link 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: ens18: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether ab:cd:ef:12:34:56 brd ff:ff:ff:ff:ff:ff altname enp0s18 To only show one interface, specify it:\nip link show ens18\nShow statistics on a particular link, which is great to learn about errors and dropped packets.\n# ip -stats link show ens18 2: ens18: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether ab:cd:ef:12:34:56 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped missed mcast 5108603572 20814332 0 789579 0 0 TX: bytes packets errors dropped carrier collsns 7504563480 10310495 0 0 0 0 altname enp0s18 Disable or enable an interface To enable an interface, use the \u0026lsquo;up\u0026rsquo; flag.\nip link set ens18 up\nTo disable, the \u0026lsquo;down\u0026rsquo; flag.\nip link set ens18 down\nChange your MAC address Besides viewing information, the subcommands can also be used to make changes, like defining a new MAC address.\nip link set dev ens18 address aa:bb:cc:dd:ee:ff\nARP cache To see the other devices on the same network segment, the neighbour replaces the functionality of the arp command.\nShow ARP cache Just use neighbour (or n) to see the ARP cache.\n# ip neighbour 192.168.1.1 dev ens18 lladdr ab:cd:ef:12:34:56 STALE Delete an ARP entry To delete an entry listed with the neighbour subcommand, define the address and interface.\nip neigh del 192.168.1.1 dev ens18\nIP and addressing Most modern systems use IP to communicate with other systems. With the subcommand address the details regarding IP can be displayed, such as active IP addresses.\nShow assigned IP address of the system.\nip address\nSingle device, which can come in handy with many aliases or VLANs.\nip address dev ens18\nBy type Limit the output by specifying its type, such as a bridge or VLAN.\nip address show type bridge\nFor VLAN tagged interfaces:\nip address show type vlan\nMulticast IP addresses # ip maddr 1:\tlo inet 224.0.0.1 inet6 ff02::1 inet6 ff01::1 2:\tens18 link 33:33:00:00:00:01 link 01:00:5e:00:00:01 link 33:33:ff:11:22:33 link 01:80:c2:00:00:00 link 01:80:c2:00:00:03 link 01:80:c2:00:00:0e inet 224.0.0.1 inet6 ff02::1:ff11:2233 inet6 ff02::1 users 2 inet6 ff01::1 TCP cache and metrics The kernel maintains a cache of entries related to TCP connections. This cache can be displayed using the subcommand tcp_metrics. Great to see recent connections with devices outside the local network.\n# ip tcp_metrics 91.92.93.94 age 433514.256sec cwnd 10 rtt 83061us rttvar 83061us source 192.168.1.123 213.212.211.210 age 75533.084sec cwnd 10 rtt 10746us rttvar 6480us source 192.168.1.123 142.143.144.145 age 9.396sec cwnd 10 rtt 9642us rttvar 9642us source 192.168.1.123 Routing table Show network routing information To find the default gateway on the network, use the route subcommand.\n# ip route default via 192.168.1.1 dev ens18 proto static 192.168.1.0/24 dev ens18 proto kernel scope link src 192.168.1.123 Test routing for a specific IP address # ip route get 192.168.2.123 192.168.2.123 via 192.168.1.1 dev ens18 src 192.168.1.123 uid 0 cache Add a route Define a default route on the ens18 interface.\nip route add default via 192.168.1.1 dev ens18\nAll traffic for our network should go via this newly defined gateway.\nip route add 192.168.1.0/24 via 192.168.1.1\nDelete route Delete a route for the defined network\nip route delete 192.168.1.0/24 via 192.168.1.1\nTips for improving default output Colored output Depending on the terminal, the colors might not be displayed by default. Enforce colors with the option -colors. It will highlight MAC addresses, IP addresses, interface status, and more.\nip -colored=always link\nSince \u0026lsquo;always\u0026rsquo; is the default, you can simplify and shorten this command:\nip -c link\nBrief output Less is more. Use -brief in a variety of subcommands.\n# ip -brief link lo UNKNOWN 00:00:00:00:00:00 \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; ens18 UP ab:cd:ef:12:34:56 \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; Brief output for IP addresses:\n# ip -brief addr lo UNKNOWN 127.0.0.1/8 ::1/128 ens18 UP 192.168.1.123/24 fe80::be24:11ff:abcd:1234/64 Brief output to show ARP entries:\n# ip -brief neighbour 192.168.1.1 ens18 12:34:56:ab:cd:ef 192.168.1.2 ens18 ab:cd:ef:12:34:56 Combining options and using columns Sometimes the output may not look as good, like misaligned or lacking clarity. Combine the brief and colored options together with the column command to align all columns.\nip -br -c link | column -t\nJSON For automated processing of data, the option -json can be added before the subcommand. Combine it with jq to filter out exactly the information that you want.\nDo you have other good ip one-liners that everyone should know?\n","permalink":"https://linux-audit.com/cheat-sheets/ip/","tags":["arp","cheatsheet","command-line","howto","ifconfig","linux","networking","one-liner","terminal"],"title":"ip cheat sheet"},{"categories":["Network","System Administration"],"contents":"The ip command can help with discovering the default gateway on a Linux system.\nList the routing table Using the -stats combined with the link subcommand, we can find statistics on a network link. This way we can see on a particular network interface how many errors or dropped packets it has.\n# ip -stats link show ens18 2: ens18: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether ab:cd:ef:12:34:56 ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped missed mcast 5108603572 20814332 0 789579 0 0 TX: bytes packets errors dropped carrier collsns 7504563480 10310495 0 0 0 0 altname enp0s18 This interface has no errors, but there were packets dropped.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-errors-and-dropped-packets-on-network-interface/","tags":["faq","howto","linux","network","statistics"],"title":"How to see errors and dropped packets on a network interface on Linux"},{"categories":["Network","System Administration"],"contents":"The ip command can help with discovering the default gateway on a Linux system.\nList the routing table Using the route subcommand, we can retrieve or configure routing information on the system. By just specifying this subcommand, the routing table will be displayed.\n# ip route default via 192.168.1.1 dev ens18 proto static 192.168.1.0/24 dev ens18 proto kernel scope link src 192.168.1.150 The default gateway on this system is 192.168.1.1.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-default-gateway/","tags":["faq","howto","network"],"title":"How to see the default gateway on Linux"},{"categories":["Network","System Administration"],"contents":"The ss command can solve the question what process is keeping a port in use.\nShow services in listening state and the related process To find services that are listening, we use the --listening option. For TCP that means the LISTEN state, UNCONN for UDP.\nAdditionally, we specify the option --processes to show the process information and --numeric to avoid resolving hostnames or service names. This way we see port numbers instead of their names.\nss --listening --numeric --processes sport :443 Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:((\u0026#34;nginx\u0026#34;,pid=344368,fd=19),(\u0026#34;nginx\u0026#34;,pid=344367,fd=19),(\u0026#34;nginx\u0026#34;,pid=344264,fd=19)) tcp LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:((\u0026#34;nginx\u0026#34;,pid=344368,fd=13),(\u0026#34;nginx\u0026#34;,pid=344367,fd=13),(\u0026#34;nginx\u0026#34;,pid=344264,fd=13)) tcp LISTEN 0 511 [::]:443 [::]:* users:((\u0026#34;nginx\u0026#34;,pid=344368,fd=14),(\u0026#34;nginx\u0026#34;,pid=344367,fd=14),(\u0026#34;nginx\u0026#34;,pid=344264,fd=14)) tcp LISTEN 0 511 [::]:443 [::]:* users:((\u0026#34;nginx\u0026#34;,pid=344368,fd=20),(\u0026#34;nginx\u0026#34;,pid=344367,fd=20),(\u0026#34;nginx\u0026#34;,pid=344264,fd=20)) In this example it is nginx that is using port 443 to listen for incoming connections. Not really surprising for web server software to serve HTTPS on port 443.\nThe shortened version:\nss -lnp sport :443\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-which-process-is-using-a-port/","tags":["faq","howto","network"],"title":"How to see which process is using a port"},{"categories":["System Administration"],"contents":"Files are an important part of Linux, as even devices and network connections are having file descriptors. The lsof command is one of the most useful tools that help with system administration, but especially with troubleshooting issues. This lsof cheat sheet tries to cover the most useful functionality and options, while trying to avoid overwhelming you like the man page might do.\nGood to know Lsof can be used as a normal user and superuser (root or with sudo permissions). Sometimes you may need superuser right to see any output at all, or have access to all information.\nCommon options Option What the option does -a Consider all selectors to be \u0026lsquo;AND\u0026rsquo; to reduce output -c Match by process name -i Match by internet address, or in other words, network connections -i4 Limit to IPv4 -i6 Limit to IPv6 -n Do not resolve IP addresses to hostnames -P Do not resolve port numbers to service names -t Show only the PIDs -u Limit files to a specific user Creating a shell script? Then we suggest using the long format option, as this improves the readability. For quick use of on the command-line consider using the short notation of the related option.\nPerforming specific requests Open files in a specific directory Want to see what is opened within a directory, use the \u0026#43;D option.\nlsof +D /var/log/journal\nBy mount point Sometimes it may be useful to learn what files are open to a specific mount point, like a NAS. Use the \u0026#43;f followed by two dashes, to signal lsof that a specific path is coming.\nlsof +f -- /mnt/backup/\nNFS mounts When using NFS, we can query all open files.\nlsof -N\nOpen files by process id (PID) lsof -p 1234\nOpen files by process name lsof -c nginx\nOpen files by user On a web server you may want to check what files are opened for the user account that runs the web server.\nlsof -u www-data\nTo see all open files, except those by root, use the caret before the username.\nlsof -u ^root\nDeleted files Sometimes files can be opened, even though they are already deleted. To uncover them, use lsof with the option \u0026#43;L1. The \u0026lsquo;1\u0026rsquo; defines the boundary, so all files with less than 1 link are shown.\nlsof +L1\nOpen network connections As everything is a file on Linux, the related sockets are also visible as files. To query them, select one or more ports.\nlsof -i :80,443\nAnother way is by protocol.\nlsof -i UDP -i TCP\nTo only show listening TCP services, limit the output. To avoid any lookups (hostname or service name), add -n and -P.\nlsof -n -P -i TCP -sTCP:Listen\nOpen connections for a single IP address can be specified as well, optionally with a port.\nlsof -n -P -i @192.168.1.1:22\nCombine options Use the -a option to tell lsof to combine the options (logical AND operator). For example to show only files (regular and directory) that are opened by the nginx process, we can combine it to a command like this.\nlsof -a -c nginx -r 3 /\nBy defining the root path, only normal files are displayed. That is, if they are opened by the process with the name \u0026rsquo;nginx\u0026rsquo;. The -r defines a refresh rate, so that we can see the open files by nginx at a given moment in time.\nSpecial use-cases Terse output to kill related processes Use the -t option to show a terse output. This means only the process IDs (PIDs) will be displayed. This can be used then as input for a command like kill.\n","permalink":"https://linux-audit.com/cheat-sheets/lsof/","tags":["cheatsheet","command-line","howto","linux","one-liner","sockets","terminal"],"title":"lsof cheat sheet"},{"categories":["Network","System Administration"],"contents":"The ss command is a great utility on Linux to see socket statistics, including opened network ports. As the tool has many options, it may not be very obvious at first what options to use. In this article we look at showing all available TCP/UDP ports that are in a listening state (TCP) or are opened (UDP) for incoming connections.\nShow TCP/UDP and related process ss -plunt\nSo what does this command do?\nOption Action performed -l State should be \u0026rsquo;listening\u0026rsquo; (TCP) -n Numeric display, do not map service names (e.g. show 22 instead of ssh) -p Include the process that opened the port -t Show TCP ports -u Show UDP port Want to see a little bit more information, including established connections? Consider using the easy to remember -plants option. Besides the established connections, it will also display a small summary at the top about the used network protocols and their state.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-open-ports-on-linux/","tags":["faq","howto","network"],"title":"How to see open ports on Linux"},{"categories":["Network","System Administration"],"contents":"Ss is the name of the tool that is replacing the netstat command. It is short for socket statistics and a great utility to show information about sockets on Linux systems. It can be used to show which TCP/UDP ports are opened or what services are listening. The only downside is that it has many options, so it may not always be clear which one to use. With this cheat sheet that should be solved!\nCommon options The ss command has many options available. Here are the most common ones that one might use during daily system administration or troubleshooting.\nLong option Short option Related action --all -a Show both listening and non-listening sockets --events -E Show sockets that are destroyed (closed connections) --info -i Internal TCP information --ipv4 -4 Only IPv4 sockets displayed --ipv6 -6 Only IPv6 sockets displayed --listening -l Only show listening sockets --no-header -H Do not show the header, great for one-liners and parsing the output --numeric -n Numeric output, conversion of names (services, ports) is skipped --processes -p Show related process that interacts with the socket --resolve -r Try resolving numeric values for addresses and ports --summary -s Display a summary with statistics at the top --tcp -t Show TCP sockets --udp -u Show UDP sockets Creating a shell script? Then using the long format option is suggested, as this improves readability and understanding what related action is may perform. For quick use of on the command-line consider using the short notation of the related option.\nNever used ss before? Run the following command to get a first good impression of the details.\nss -plants\nThis set of option is easy to remember and shows many useful insights. It includes:\nSummary All connections Does show port and service numbers instead of names Includes process names Query specific types of connections With the help of expressions, we can filter the data and display the specific information that we are looking for.\nPredicate Operator Filter unit Explanation autobound Match if port or path was automatically allocated cgroup = or != PATH Match by path and connection is (not) part of cgroup dev = or != DEVICE Match by device (or not) dst = HOST Destination equals a specific host or network src = HOST Source equals a specific host or network dport \u0026ldquo;\u0026lt;\u0026rdquo;, \u0026ldquo;\u0026lt;=\u0026rdquo;, \u0026ldquo;=\u0026rdquo;, \u0026ldquo;!=\u0026rdquo;, \u0026ldquo;\u0026gt;=\u0026rdquo;, or \u0026ldquo;\u0026gt;\u0026rdquo; [FAMILY:]:PORT Match by destination port sport \u0026ldquo;\u0026lt;\u0026rdquo;, \u0026ldquo;\u0026lt;=\u0026rdquo;, \u0026ldquo;=\u0026rdquo;, \u0026ldquo;!=\u0026rdquo;, \u0026ldquo;\u0026gt;=\u0026rdquo;, or \u0026ldquo;\u0026gt;\u0026rdquo; [FAMILY:]:PORT Match by source port FAMILY is one of values: unix, inet, inet6, link, netlink, vsock, tipc, xdp\nBy port number On a web server it makes sense to see the open connections on HTTPS (port 443).\nss -nt sport = :443\nTo query multiple ports\nss -nt '( sport = :443 or sport = :80 )'\nA slightly shorter version is by defining the side \u0026lsquo;src\u0026rsquo; (source) or \u0026lsquo;dst\u0026rsquo; (destination)\nss -nt '( src :443 or src :80 )'\nBy destination To see active connections with a specific destination, define an expression including the IP address or address. For example to see connections on the 192.168.x.x network:\nss dst 192.168/16\nQuery specific details See connection and transmission speed The --info option reveals a lot of specifics, including the send and receive speed. Interesting fields\nsend pacing_rate delivery_rate # ss --info dst 192.168.1.11 dport 2049 Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp ESTAB 0 0 192.168.1.10:682 192.168.1.11:nfs cubic wscale:9,7 rto:204 rtt:0.23/0.063 ato:40 mss:1448 pmtu:1500 rcvmss:1448 advmss:1448 cwnd:10 ssthresh:417 bytes_sent:594656932 bytes_retrans:1448 bytes_acked:594655485 bytes_received:347264820 segs_out:1116447 segs_in:852268 data_segs_out:1033392 data_segs_in:765849 send 504Mbps lastsnd:34580 lastrcv:34580 lastack:4364 pacing_rate 1.01Gbps delivery_rate 895Mbps delivered:1033393 busy:310384ms retrans:0/1 dsack_dups:1 reordering:7 reord_seen:1299 rcv_rtt:2.256 rcv_space:266348 rcv_ssthresh:1215980 minrtt:0.121 TLS/SSL version and Ciphers Some protocol specifics can be displayed as well. In this example we see TLSv1.3 with the cipher AES-GCM-256 being used.\n# ss -piment ESTAB 0 0 11.22.33.44:443 55.66.77.88:37912 users:((\u0026#34;nginx\u0026#34;,pid=342995,fd=5)) uid:33 ino:28900680 sk:97 cgroup:/system.slice/nginx.service \u0026lt;-\u0026gt; skmem:(r0,rb131072,t0,tb4194304,f0,w0,o0,bl0,d0) ts sack cubic wscale:9,7 rto:204 rtt:0.286/0.082 ato:40 mss:1448 pmtu:1500 rcvmss:666 advmss:1448 cwnd:19 bytes_sent:14455 bytes_acked:14455 bytes_received:1261 segs_out:17 segs_in:12 data_segs_out:14 data_segs_in:4 send 769566434bps lastsnd:15744 lastrcv:15680 lastack:15680 pacing_rate 1538460456bps delivery_rate 260640000bps delivered:15 app_limited reordering:254 rcv_space:14600 rcv_ssthresh:64076 minrtt:0.18 snd_wnd:60928 tcp-ulp-tls version: 1.3 cipher: aes-gcm-256 rxconf: none txconf: sw Tip: remember this set as options as \u0026lsquo;pigment\u0026rsquo; without the g.\nMonitoring connections To see if there is traffic on a system, use the --events option. It will display the sockets that are destroyed. Or in other words, the connections that are closed. A great way to see the amount of traffic and great for monitoring or when to do system maintenance.\nss -n --events\nManually close a connection The ss command can also be used to close active connections. It works for IPv4 and IPv6 and can be used with the --kill option. Typically you want to combine this with a specific IP address and optionally a port.\nss --kill dst 192.168.1.123 dport = 80\nSee timer information Some services like SSH want to stay connected. They send a keepalive signal now and then to keep the connection active. For TCP connections, we can request timer information and see when a timer expires.\nss --options --tcp\nThe value displayed after \u0026lsquo;keepalive\u0026rsquo; refers to the expiry time. So when renewing it, the values typically go down.\n","permalink":"https://linux-audit.com/cheat-sheets/ss/","tags":["cheatsheet","howto","linux","networking","one-liner","sockets"],"title":"ss cheat sheet"},{"categories":["Network","System Administration"],"contents":"The TTL value defines how long a DNS record normally should be cached. Although not all resolvers will adhere to it, many do. In that case it may be useful to learn how much time is still left on the TTL.\nUsing dig to query the TTL With the dig command we query the current TTL value. That is, the value returned by your DNS resolver. When the value reaches zero, a new request should be done to the upstream servers, often the authoritative server.\ndig +noall +answer +ttlunits A linux-audit.com\nSo what does this command do?\nDig option Action performed +noall Filter results, don\u0026rsquo;t show all details of the query response +answer Include the actual answer in the output +ttlunits Show a human friendly output, instead of seconds For readability we normally list all options in alphabetical order. For this command it is important that \u0026#43;answer comes after \u0026#43;noall, otherwise the answer will be filtered out.\n","permalink":"https://linux-audit.com/networking/faq/how-to-see-the-ttl-of-a-dns-record/","tags":["dns","faq","howto","network"],"title":"How to see the TTL value of a DNS record"},{"categories":["Software","System Administration"],"contents":"Similar to Debian systems, Ubuntu uses apt and dpkg to do basic package management. The Debian package manager, or dpkg, is available for the task to show all installed packages.\nUsing dpkg to list packages Running the dpkg command is as simple as providing only the providing the --list option.\n# dpkg --list Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-======================================-=======================================-============-================================================================================ ii adduser 3.118ubuntu5 all add and remove users and groups ii amd64-microcode 3.20191218.1ubuntu2.2 amd64 Processor microcode firmware for AMD CPUs ii apparmor 3.0.4-2ubuntu2.3 amd64 user-space parser utility for AppArmor ii apport 2.20.11-0ubuntu82.5 all automatically generate crash reports for debugging ii apport-symptoms 0.24 all symptom scripts for apport ii apt 2.4.12 amd64 commandline package manager ii apt-utils 2.4.12 amd64 package management related utility programs ii base-files 12ubuntu4.6 amd64 Debian base system miscellaneous files ii base-passwd 3.5.52build1 amd64 Debian base system master password and group files ","permalink":"https://linux-audit.com/software/package-manager/faq/show-installed-packages-on-ubuntu/","tags":["dpkg","faq","howto","linux","package manager","software","ubuntu"],"title":"How to show all installed packages on Ubuntu"},{"categories":["Software","System Administration"],"contents":" Linux distribution Full command Short command Alternative Alma Linux dnf list installed Debian dpkg \u0026ndash;list dpkg -l openSUSE zypper search \u0026ndash;installed-only zypper search -i Ubuntu dpkg \u0026ndash;list dpkg -l Alma Linux, Fedora, RHEL, RockyLinux Systems that are based on RHEL or are similar, have the option to use dnf command, which will show installed packages.\ndnf list installed\nDebian and Ubuntu Systems that run or based on Debian or Ubuntu can use the dpkg command to list the installed packages.\ndpkg --list OpenSUSE and SUSE For OpenSUSE and SUSE systems, the zypper command can show the installed packages. Use the search subcommand with the --installed-only option.\nzypper search --installed-only ","permalink":"https://linux-audit.com/software/package-manager/show-installed-packages-on-linux/","tags":["dpkg","howto","linux","package manager","software","zypper"],"title":"List installed packages on a Linux system"},{"categories":["Hardware","System Administration"],"contents":"Most modern hardware has USB support. Therefore most Linux kernels have modular support for USB devices as well, making it easy to plug a device and get started with it. Sometimes you may need to retrieve the details and then it is useful to know how to query the available USB devices and the details.\nShow list of connected USB devices To show available USB devices, the lsusb command is a good start.\nlsusb\nRetrieving more details Combine the number of lines and specify a unit to see more relevant entries.\n# lsusb --tree /: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M |__ Port 3: Dev 4, If 0, Class=Video, Driver=uvcvideo, 5000M |__ Port 3: Dev 4, If 1, Class=Video, Driver=uvcvideo, 5000M |__ Port 3: Dev 4, If 2, Class=Audio, Driver=snd-usb-audio, 5000M |__ Port 3: Dev 4, If 3, Class=Audio, Driver=snd-usb-audio, 5000M |__ Port 3: Dev 4, If 4, Class=Human Interface Device, Driver=usbhid, 5000M |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/2p, 10000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M |__ Port 4: Dev 9, If 0, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 2: Dev 8, If 1, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 2: Dev 8, If 0, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 1: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 480M |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/2p, 480M |__ Port 2: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 4: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 4: Dev 4, If 2, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 4: Dev 4, If 3, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 4: Dev 4, If 4, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 10: Dev 7, If 0, Class=Wireless, Driver=btusb, 12M |__ Port 10: Dev 7, If 1, Class=Wireless, Driver=btusb, 12M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M To get more details, add the --verbose option.\n/sys/bus/usb/devices Another option is to look in the directories located under /sys/bus/usb/devices. A numbered set of directories lists each devices and all its available parameters.\n# ls -l /sys/bus/usb/devices/usb1/ total 0 drwxr-xr-x 6 root root 0 Apr 17 19:08 1-0:1.0 drwxr-xr-x 5 root root 0 Apr 17 19:08 1-1 -rw-r--r-- 1 root root 4096 Apr 28 22:55 authorized -rw-r--r-- 1 root root 4096 Apr 28 22:55 authorized_default -rw-r--r-- 1 root root 4096 Apr 28 22:55 avoid_reset_quirk -rw-r--r-- 1 root root 4096 Apr 17 19:08 bConfigurationValue -r--r--r-- 1 root root 4096 Apr 17 19:08 bDeviceClass -r--r--r-- 1 root root 4096 Apr 28 22:55 bDeviceProtocol -r--r--r-- 1 root root 4096 Apr 28 22:55 bDeviceSubClass -r--r--r-- 1 root root 4096 Apr 28 22:55 bMaxPacketSize0 -r--r--r-- 1 root root 4096 Apr 28 22:55 bMaxPower -r--r--r-- 1 root root 4096 Apr 28 22:55 bNumConfigurations -r--r--r-- 1 root root 4096 Apr 28 22:55 bNumInterfaces -r--r--r-- 1 root root 4096 Apr 17 19:08 bcdDevice -r--r--r-- 1 root root 4096 Apr 28 22:55 bmAttributes -r--r--r-- 1 root root 4096 Apr 17 19:08 busnum -r--r--r-- 1 root root 4096 Apr 28 22:55 configuration -r--r--r-- 1 root root 65553 Apr 17 19:08 descriptors -r--r--r-- 1 root root 4096 Apr 28 22:55 dev -r--r--r-- 1 root root 4096 Apr 17 19:08 devnum -r--r--r-- 1 root root 4096 Apr 28 22:55 devpath lrwxrwxrwx 1 root root 0 Apr 17 19:08 driver -\u0026gt; ../../../../bus/usb/drivers/usb drwxr-xr-x 3 root root 0 Apr 17 19:08 ep_00 -r--r--r-- 1 root root 4096 Apr 17 19:08 idProduct -r--r--r-- 1 root root 4096 Apr 17 19:08 idVendor -rw-r--r-- 1 root root 4096 Apr 28 22:55 interface_authorized_default -r--r--r-- 1 root root 4096 Apr 28 22:55 ltm_capable -r--r--r-- 1 root root 4096 Apr 17 19:08 manufacturer -r--r--r-- 1 root root 4096 Apr 28 22:55 maxchild drwxr-xr-x 2 root root 0 Apr 17 19:08 power -r--r--r-- 1 root root 4096 Apr 17 19:08 product -r--r--r-- 1 root root 4096 Apr 28 22:55 quirks -r--r--r-- 1 root root 4096 Apr 28 22:55 removable --w------- 1 root root 4096 Apr 28 22:55 remove -r--r--r-- 1 root root 4096 Apr 28 22:55 rx_lanes -r--r--r-- 1 root root 4096 Apr 17 19:08 serial -r--r--r-- 1 root root 4096 Apr 17 19:08 speed lrwxrwxrwx 1 root root 0 Apr 26 12:36 subsystem -\u0026gt; ../../../../bus/usb -r--r--r-- 1 root root 4096 Apr 28 22:55 tx_lanes -rw-r--r-- 1 root root 4096 Apr 26 12:36 uevent -r--r--r-- 1 root root 4096 Apr 28 22:55 urbnum -r--r--r-- 1 root root 4096 Apr 28 22:55 version Monitor using udevadm For troubleshooting or discovering a specific device, there might be a better way. With the help of udevadm we can tell it to monitor for events related to the USB subsystem. This makes it easy to learn more about a particular device by plugging it in, or removing it again.\n# udevadm monitor --subsystem-match=usb --udev monitor will print the received events for: UDEV - the event which udev sends out after rule processing UDEV [1203685.522963] add /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1 (usb) UDEV [1203685.526742] add /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.0 (usb) UDEV [1203685.529552] add /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.1 (usb) UDEV [1203685.547246] bind /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.1 (usb) UDEV [1203685.602013] bind /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.0 (usb) UDEV [1203685.606676] bind /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1 (usb) UDEV [1203699.839437] unbind /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.0 (usb) UDEV [1203699.840795] remove /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.0 (usb) UDEV [1203699.885635] unbind /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.1 (usb) UDEV [1203699.886440] remove /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1/3-2.1:1.1 (usb) UDEV [1203699.887547] unbind /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1 (usb) UDEV [1203699.888441] remove /devices/pci0000:00/0000:00:14.0/usb3/3-2/3-2.1 (usb) In this example we see add and bind events when the device is plugged in. After removal, the unbind and remove events show up.\n","permalink":"https://linux-audit.com/hardware/faq/how-to-list-all-usb-devices/","tags":["faq","hardware","howto","usb"],"title":"How to list all USB devices"},{"categories":["Hardware","System Administration"],"contents":"Linux makes it fairly easy to show the details of the BIOS and other hardware components. One option to retrieve hardware details, is using the dmidecode tool. By specifying the type and set this to bios, we can pull in the basics of the BIOS. Let\u0026rsquo;s have a look at some examples.\nShow basic BIOS details Use --typebios to query BIOS details.\n# dmidecode --type bios # dmidecode 3.2 Getting SMBIOS data from sysfs. SMBIOS 3.1.1 present. Handle 0x0000, DMI type 0, 26 bytes BIOS Information Vendor: Intel Corp. Version: HNKBLi70.86A.0096.2024.0910.2941 Release Date: 03/10/2021 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported 5.25\u0026#34;/1.2 MB floppy services are supported (int 13h) 3.5\u0026#34;/720 kB floppy services are supported (int 13h) 3.5\u0026#34;/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 5.6 Handle 0x003C, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Long Installable Languages: 1 en|US|iso8859-1 Currently Installed Language: en|US|iso8859-1 Motherboard information To see the details about the motherboard, which may reveal more about the hardware itself, we can request the baseboard type.\n# dmidecode --type baseboard # dmidecode 3.2 Getting SMBIOS data from sysfs. SMBIOS 3.1.1 present. Handle 0x0002, DMI type 2, 15 bytes Base Board Information Manufacturer: Intel Corporation Product Name: NUC8i7HVB Version: J68196-602 Serial Number: BTHN000008AB Asset Tag: Features: Board is a hosting board Board is replaceable Location In Chassis: Default string Chassis Handle: 0x0003 Type: Motherboard Contained Object Handles: 0 Handle 0x0028, DMI type 41, 11 bytes Onboard Device Reference Designation: AMD HD Graphics Device Type: Video Status: Enabled Type Instance: 1 Bus Address: 0000:00:01.0 More information types By specifying the type, we can get information about the system itself.\nValid options include:\nbaseboard bios cache chassis connector memory processor slot system ","permalink":"https://linux-audit.com/hardware/faq/how-to-see-bios-details/","tags":["bios","faq","hardware","howto","linux"],"title":"How to see BIOS details on Linux?"},{"categories":["Hardware","System Administration"],"contents":"Under normal conditions, you may not need to know much about the hard disk(s) itself. But when you do, it is nice to know what tools are available to query the right piece of information. Information like vendor, disk type, speed, and size.\nShow available hard disk devices If we want to know what hard disks are available, have a look at how to see the available hard disks.\nBasic information about hard disks (using lshw) Basic information about a hard disk can be displayed using lshw. It shows the device name, size, sector sizes, and some capabilities of the hard disk. It\u0026rsquo;s a great start, and sometimes enough to get the information you wanted.\n# lshw -class disk *-cdrom description: DVD reader product: QEMU DVD-ROM vendor: QEMU physical id: 0.0.0 bus info: scsi@1:0.0.0 logical name: /dev/cdrom logical name: /dev/sr0 version: 2.5+ capabilities: removable audio dvd configuration: ansiversion=5 status=nodisc *-disk description: SCSI Disk product: QEMU HARDDISK vendor: QEMU physical id: 0.0.0 bus info: scsi@2:0.0.0 logical name: /dev/sda version: 2.5+ size: 16GiB (17GB) capabilities: 5400rpm gpt-1.00 partitioned partitioned:gpt configuration: ansiversion=5 guid=59907fc9-3694-4808-8f5b-9032ae1b279d logicalsectorsize=512 sectorsize=512 Show specifics about SATA hard disks (using hdparm) The next tool is hdparm and like previous tool often available on Linux systems. To query identification information from the drive itself, we can use the -I option.\n# hdparm -I /dev/sda /dev/sda: ATA device, with non-removable media Standards: Likely used: 1 Configuration: Logical\tmax\tcurrent cylinders\t0\t0 heads\t0\t0 sectors/track\t0\t0 -- Logical/Physical Sector size: 512 bytes device size with M = 1024*1024: 0 MBytes device size with M = 1000*1000: 0 MBytes cache/buffer size = unknown Capabilities: IORDY not likely Cannot perform double-word IO R/W multiple sector transfer: not supported DMA: not supported PIO: pio0 Information about NVMe devices (using nvme) The hdparm command will most likely not able to show information about newer disk types. For NVMe disks, we can use the nvme tool.\n# nvme list /dev/nvme0 Node SN Model Namespace Usage Format FW Rev --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 SSSSSSSSSSSSSSS Samsung SSD 960 PRO 1TB 1 89,74 GB / 1,00 TB 512 B + 0 B 12345678 /dev/nvme1n1 STTTTTTTTTTTTTT Samsung SSD 960 PRO 1TB 1 53,20 GB / 1,00 TB 512 B + 0 B 12345678 Important: without root permissions, no details will be shown\nThe nvme tool is most likely part of the nvme-cli package and might to be installed first.\n","permalink":"https://linux-audit.com/hardware/faq/how-to-see-hard-disk-specifications-and-details/","tags":["faq","hardware","howto"],"title":"How to see hard disk specifications and details"},{"categories":["Hardware","System Administration"],"contents":"Show available hard disk devices Using blkid The first tool to query the available disks is blkid. It shows a list of devices and is available on most Linux distributions.\nblkid\nUsing lsblk Another great utility is lsblk. As the name implies, it shows block devices, like a hard disk. It will show a tree-like structure with the basic details. Its output is easier to read than that of blkid.\n# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 87M 1 loop /snap/lxd/27948 loop1 7:1 0 87M 1 loop /snap/lxd/28373 loop2 7:2 0 39.1M 1 loop /snap/snapd/21184 loop4 7:4 0 63.9M 1 loop /snap/core20/2182 loop5 7:5 0 63.9M 1 loop /snap/core20/2264 loop6 7:6 0 38.7M 1 loop /snap/snapd/21465 sda 8:0 0 16G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1.8G 0 part /boot └─sda3 8:3 0 14.2G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 14.2G 0 lvm / sr0 11:0 1 1024M 0 rom In this output we can see that we have one disk (sda) that contains multiple partitions (sda{1,2,3}). A valuable piece of information is the major device identifier (8). It is a block device and the shortened name is sd, which stands for SCSI disk devices. For NVMe devices, the major device ID is 259.\nIf we want to query which hard disks are available, we can do an alternative query using a filter with the major device IDs 8 and 259.\nlsblk --nodeps --noheadings --include 8,259 --output NAME\nHere we ask lsblk to show us only the name of the devices with major type 8 or 259. No dependencies and no headings in the output.\nQuery disk devices using the lshw command Another great tool is lshw and lists hardware. By defining the class to disk, we see hard drivers and other disk devices.\n# lshw -class disk *-cdrom description: DVD reader product: QEMU DVD-ROM vendor: QEMU physical id: 0.0.0 bus info: scsi@1:0.0.0 logical name: /dev/cdrom logical name: /dev/sr0 version: 2.5+ capabilities: removable audio dvd configuration: ansiversion=5 status=nodisc *-disk description: SCSI Disk product: QEMU HARDDISK vendor: QEMU physical id: 0.0.0 bus info: scsi@2:0.0.0 logical name: /dev/sda version: 2.5+ size: 16GiB (17GB) capabilities: 5400rpm gpt-1.00 partitioned partitioned:gpt configuration: ansiversion=5 guid=59907fc9-3694-4808-8f5b-9032ae1b279d logicalsectorsize=512 sectorsize=512 ","permalink":"https://linux-audit.com/hardware/faq/how-to-see-the-available-hard-disks/","tags":["faq","hardware","howto"],"title":"How to see the available hard disks"},{"categories":["System Administration"],"contents":"Basic options Long option Short option What the option does --human-readable -h Show size in human readable format, such as kilobytes and megabytes --max-depth=NUMBER -d Set a maximum depth to summary, useful to see size of a particular directory --summarize -s Don\u0026rsquo;t show all files or directories, but summarize the output --threshold=-VAL -t Files smaller than VAL --time Show the last modification time of any file or a sub directory in the specified directory Creating a shell script? Then we suggest using the long format option, as this improves the readability. For quick use of on the command-line consider using the short notation of the related option.\nShow size for a single directory By setting a max-depth of zero, we can see the size of a particular directory.\ndu --human-readable --max-depth=0 /var\nAn alternative is using the --summarize\nFiles by a minimum size Only see files smaller than 1 megabyte.\ndu --human-readable --threshold=-1M /var\nTo show all files bigger than a specified size, use the threshold with a positive number. For example, all files bigger than 10 megabyte:\ndu --human-readable --threshold=10M /var\n","permalink":"https://linux-audit.com/cheat-sheets/du/","tags":["cheatsheet","howto","one-liner"],"title":"du cheat sheet"},{"categories":["System Administration"],"contents":"Looking to clean up some data, but unclear what the biggest directories are on the disk or within a partition? In this FAQ we look at how to quickly find them.\nFind biggest directories With the du command we can count the disk usage of directories. By using the --all option, it will combine the size of both files and underlying directories.\nThere is also the option --one-file-system, which will count only directories and files on the same file system. This is useful when there are external mounts. For example when using NFS, we don\u0026rsquo;t want to scan the full NAS as well.\ndu --all --one-file-system /\nWant to use the shortened options and sort the output? Use sort to do the sorting and head to only show 10 results.\ndu -a -x / | sort -n -r | head\nCombine the options of the commands to make them even shorter.\ndu -ax | sort -nr | head\nMaking it human-readable The output can be a bit friendlier by using human-readable output. In that case, we need to tell both du and sort to change the default output style.\ndu --all --human-readable --one-file-system / | sort --human-numeric-sort --reverse | head\nWant to use the shortened options for this command as well?\ndu -ahx / | sort -hr | head\n","permalink":"https://linux-audit.com/filesystems/faq/how-to-find-the-biggest-directories-on-disk/","tags":["faq","file system","howto","linux","sort"],"title":"How to find the biggest directories on disk"},{"categories":["System Administration"],"contents":"Systemd uses the concept of masked units that prevents those units from being started. This can be used for one-time tasks, like those that need to be executed only the first time after the installation. Another reason is that a system administrator might want to disable an unused service or one that is being tested.\nShow masked units To show the masked units, we can ask systemctl to show all unit files with a state of masked.\nsystemctl list-unit-files --state=masked\nWhen there are matches, the output could look like this:\n# systemctl list-unit-files --state=masked UNIT FILE STATE PRESET cryptdisks-early.service masked enabled cryptdisks.service masked enabled hwclock.service masked enabled rc.service masked enabled rcS.service masked enabled screen-cleanup.service masked enabled sudo.service masked enabled x11-common.service masked enabled 8 unit files listed. Unmask a unit or service In most cases, a masked unit is a service. Need a particular service to run again? Use the unmask command.\nsystemctl unmask NAME.service\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-all-masked-units-with-systemctl/","tags":["faq","howto","linux","systemctl","systemd"],"title":"How to see all masked units with systemctl"},{"categories":["System Administration"],"contents":"Sometimes a systemd unit, like a service, should not be starting during boot time or not at all. Let\u0026rsquo;s have a look how to disable a systemd unit.\nSee the last few lines By default, journalctl will display all entries from the journal. To limit this to last X lines, use the -n option and optionally define a number (default: 10 lines).\njournalctl -n 5\nLimit the output for a service Combine the number of lines and specify a unit to see more relevant entries.\njournalctl -u ssh.service -n 20\nTip: if you query the status output with systemctl, it will also pull the last ten lines from the journal (similar to -n 10).\n","permalink":"https://linux-audit.com/systemd/faq/how-to-see-the-last-x-lines-with-journalctl/","tags":["faq","howto","journalctl","linux","systemd"],"title":"How to see the last X lines with journalctl"},{"categories":["System Administration"],"contents":"Sometimes a systemd unit, like a service, should not be starting during boot time or not at all. Let\u0026rsquo;s have a look how to disable a systemd unit.\nCompletely disable a unit The most strict method is to disable a unit using the mask subcommand.\nsystemctl mask UNIT\nWith the service masked, it won\u0026rsquo;t be able to start it anymore. Only when using the unmask command, it can be reactivated again.\nDisable during boot Just want to disable the unit so it won\u0026rsquo;t be started during the boot process? Then we use the disable subcommand.\nsystemctl disable UNIT\nAfter changing the service, have a look at it. In the line starting with Loaded, it should list \u0026lsquo;disabled\u0026rsquo; after the unit name.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-disable-a-systemd-unit-with-systemctl/","tags":["faq","howto","linux","systemctl","systemd"],"title":"How to disable a systemd unit with systemctl"},{"categories":["System Administration"],"contents":"Systemd units typically fail when something in the execution of a service does not go as planned. Another option is that a specific condition is not met. Instead to continue running the service, it then can also be marked as failed. In this How To we look at how to show failed units.\nShowing failed units The shortest option to get information about units that failed is using the --failed option.\nsystemctl --failed\nIf all is well, then the output may look like this:\n# systemctl list-units --state=failed UNIT LOAD ACTIVE SUB DESCRIPTION 0 loaded units listed. If a service has failed, then it will show the unit(s) and also the total count (last line).\nUNIT LOAD ACTIVE SUB DESCRIPTION ● my-failed-service.service loaded failed failed Failure test LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. Aliases that do the same as --failed include:\nsystemctl list-units --state=failed systemctl list-units --failed Testing if a unit has failed For some service it may be interesting to monitor them, especially if they have a higher chance of failing. To test if a unit has failed, we can use the is-failed command.\nsystemctl is-failed ssh.service\nThe command will return a single line with the state:\nState Exit code Meaning active 1 service is running failed 0 service has failed inactive 1 service is not active The exit code is useful for shell scripting. Use it in combination with the --quiet option to silence the output.\nif systemctl is-failed --quiet my-failed-service; then echo \u0026#34;My service failed\u0026#34; fi ","permalink":"https://linux-audit.com/systemd/faq/how-to-show-failed-units-with-systemctl/","tags":["faq","howto","linux","systemctl","systemd"],"title":"How to show failed units with systemctl"},{"categories":["System Administration"],"contents":"After the installation of a software package or when configuring a custom unit, it may be needed to start the unit. In most cases, we also want our unit to start during boot time. There are a few ways to achieve this, let\u0026rsquo;s have a look.\nManual start and enable First we start the unit using the start command.\nsystemctl start UNIT\nAfter that, we enable the unit.\nsystemctl enable UNIT\nIf everything went well, the unit is now started and will also start during the next boot.\nCombine the options There is a quicker way to achieve the steps above. This is to enable the service and start it right away using the --now option.\nsystemctl enable --now UNIT\nSame result, but in one command!\nCheck the status After starting the service, have a look at the status of the unit to see if everything is expected.\nsystemctl status UNIT\nEspecially the line in the output with Loaded and Active will reveal if the unit is running and enabled.\n","permalink":"https://linux-audit.com/systemd/faq/how-to-start-and-enable-a-unit-with-systemctl/","tags":["faq","howto","linux","systemctl","systemd"],"title":"How to start and enable a unit with systemctl"},{"categories":["System Administration"],"contents":"Basic options Find has many options, but here are the ones that are commonly used. Often in combination to gain a better search job.\nLong option Short option What the option does -exec VALUE Perform some command on the search rules -group GROUP Search by ownership (group) -print Just show the search results -size VALUE Limit by file size, with minus being smaller than, plus is bigger than specified size -type d Only search for directories -type f Only search for files -user USERNAME Search by ownership (user) -xdev Do not cross between different file systems (e.g. NFS) Creating a shell script? Use the long format option, as this improves the readability. For quick use of find on the command-line consider using the short notation of the related option.\nDirectories Searching for a directory with a specific name, can be done by specifying the type and name.\nfind / -type d -name etc\nTo find all empty directories under the current work directory, use the -empty option.\nfind . -type d -empty\nBy permissions The find command has the option to limit the search to the specific [file permissions](/filesystems/file-permissions/ of a file.\nFind files with setuid (SUID) and setgid (SGID) To see what files have the setuid bit:\nfind . -perm -4000\nAnother notation to do the same:\nfind . -perm /u=s\nSearch the setgid bit by using a \u0026lsquo;2\u0026rsquo; as the first number\nfind . -perm -2000\nSimilar to setuid, we can use an alternative notation and search for the \u0026rsquo;s\u0026rsquo; in the group:\nfind . -perm /g=s\nBy ownership: user or group To find all the files owned by a specific user, define the username.\nfind . -user michael\nAnother option is searching all files selected by a specific group.\nfind . -group adm\nBy file size Smaller than 1 megabyte:\nfind . -size -1M\nSearch files bigger than a specific size, like more than 1 megabyte.\nfind . -size +1M\nThe -size option can also be combined to find a file with a minimum size and maximum size.\nBy date of time Modification time Want to find the files for which the content was recently changed? Use the modified time and select the time in minutes:\nfind . -type f -mmin -15\nWhen looking for files that are changed for a longer period, change the minus into a plus and specify the period (e.g. older than 1 week).\nfind . -type f -mtime +1w\nModification date Looking for files that are changed after a specific date?\nfind . -type f -newermt 2024-05-01\nTo find files modified in a specific date range, set the begin and end. For example to define a specific week:\nfind . -type f -newermt 2024-05-01 ! -newermt 2024-05-08\nAccess date Like the modification date, we can search for files that are recently accessed. To define a specific day, tell find the start date and the stop date.\nfind . -type f -newerat 2024-05-08 ! -newerat 2024-05-09\nBy depth Sometimes you don\u0026rsquo;t want to go multiple levels deep into the underlying subdirectories. Specify the depth to search, so that find knows when to stop.\nfind . -maxdepth 1 -print\nApplying changes to files found Correcting file permissions Change all files that have file permissions of \u0026lsquo;777\u0026rsquo; to more a sane value of \u0026lsquo;644\u0026rsquo;:\nfind . -type f -perm 777 -print -exec chmod 644 {} \\;\nFor directories that would most likely be 755:\nfind . -type d -perm 777 -print -exec chmod 755 {} \\;\n","permalink":"https://linux-audit.com/cheat-sheets/find/","tags":["cheatsheet","howto","one-liner"],"title":"find cheat sheet"},{"categories":null,"contents":"","permalink":"https://linux-audit.com/website/sitemap/","tags":["website"],"title":"Sitemap"},{"categories":["System Administration"],"contents":"Basics Systemd is a system and service manager. It is replacing older init systems and running Process ID (PID) 1, managed by the kernel itself.\nWant to learn more? Have a look at the systemd section.\nSystemd information Command Performed action systemctl get-default Show default target (like run level) systemctl list-automounts Show automounts systemctl list-dependencies Show dependencies of an unit or default target systemctl list-jobs View active jobs systemctl list-sockets List sockets and what it activates systemctl list-timers List timers (scheduled tasks, similar to cronjobs) systemctl list-unit-files Show unit files and state systemctl list-units Show if units are loaded/active Basics for services Stop and start Command Performed action systemctl stop Stop running service systemctl start Start service systemctl restart Restart running service systemctl reload Reload config files for service Status and unit changes Command Performed action systemctl daemon-reload Reload changed unit files systemctl status Show status of service systemctl state=failed (or --failed) Show failed services systemctl reset-failed Resets unit(s) with failed state systemctl enable Enable service or unit and allow start on boot systemctl disable Disable service or unit, don\u0026rsquo;t start at boot systemctl mask Fully disable a unit systemctl unmask Reactivate unit again after being masked Configuration Command Performed action systemctl cat Show unit file details systemctl show Show properties systemctl edit Create configuration as drop in unit (extension to primary config) systemctl edit --full Edit primary unit file for service \u0026raquo; Mastering the tool: systemctl\nsystemctl cheat sheet Journal Instead of a log files, systemd stores logging information in journals.\nLong option Short option What the option does --follow -f Track changes, like tail -f --output= -o Define what output format should be used for journal entries --reverse -r Reverse output, newest on top --since -S Limit the data to a specific period \u0026raquo; Mastering the tool: journalctl\njournalctl cheat sheet System state The options to change the system state are:\nsystemctl reboot systemctl poweroff systemctl suspend systemctl hibernate ","permalink":"https://linux-audit.com/cheat-sheets/systemd/","tags":["systemd","cheatsheet","howto"],"title":"systemd cheat sheet"},{"categories":["System Administration","Web"],"contents":"The web can save a lot of traffic by using optimized caching, especially for static files such as style sheets (CSS), JavaScript, images, and many other files containing static data. One option to set up caching is using the Expires header. Next step is to actually test if caching is working as expected!\nUsing ETag header When ETag or entity tag is used on static files, we can retrieve the value from the response header. In that case we have to test it against a path that most likely is or should be cached, like an image.\n1 2 3 4 curl --compressed \\ --head \\ --no-progress-meter \\ https://example.com/logo.svg | grep -i \u0026#34;^etag:\u0026#34; | awk \u0026#39;{print $2}\u0026#39; This request will allow data encoding (1), and only show the response headers (2) without a progress meter (3). Finally we define the request itself and pull in the specific ETag header (4), followed by showing its value (second column).\nNo output? Then ETag is most likely not used or not applicable to that file type. Test it against another type of file.\nIf we do get a response, copy that into the request If-None-Match.\n1 2 3 4 5 6 7 curl --compressed \\ --head \\ --header \u0026#39;If-None-Match: \u0026#34;65fc4369-119\u0026#34;\u0026#39; \\ --no-progress-meter \\ --output /dev/null \\ --write-out \u0026#39;%{http_code}\u0026#39; \\ https://example.com/logo.svg In this example we request a compressed version (1) of a SVG file (7). We perform a HEAD request (2) and any output (4 and 5) is redirected, as that part we are not interested in. The ETag is tested with a specific value, and finally we want only to see the HTTP status code (6).\nIf caching works as expected, you should receive a three digit response (304), which stands for HTTP 304 Not Modified. If you get a 200 or other code, then caching is not working, or the file can\u0026rsquo;t be found.\nUsing Last-Modified header Another option is using the header value stored in Last-Modified. As the name implies, it tells the last modification of that particular file. Let\u0026rsquo;s see if we can retrieve it first:\n1 2 3 4 # curl --compressed \\ --head \\ --no-progress-meter \\ https://example.com/logo.svg | grep -i \u0026#34;^last-modified:\u0026#34; | awk -F\u0026#34;: \u0026#34; \u0026#39;{print $2}\u0026#39; The response might look like this\nThu, 21 Mar 2024 14:25:45 GMT Next step is using this value and combine it with the If-Modified-Since header.\n1 2 3 4 5 6 7 curl --compressed \\ --head \\ --header \u0026#39;If-Modified-Since: Thu, 21 Mar 2024 14:25:45 GMT\u0026#39; \\ --no-progress-meter \\ --output /dev/null \\ --write-out \u0026#39;%{http_code}\u0026#39; \\ https://example.com/logo.svg This is similar to the ETag test, except line three.\nGot another 304? Perfect, then caching is working.\n","permalink":"https://linux-audit.com/web/test-web-server-caching-with-curl/","tags":["awk","curl","howto","performance","web server"],"title":"Test web server caching with curl"},{"categories":["System Administration"],"contents":"Which systemd unit types are available and what is their goal? In this article we cover them and show some useful commands related to these units.\nShowing all available unit types Let\u0026rsquo;s start by displaying some of the units that are available on your system.\nsystemctl list-unit-files\nThis will reveal types like:\nAutomount Device Mount Path Scope Service Slice Snapshot Socket Target Timer Retrieve basic information from a unit Before we dive into all unit types, it is good to know a few commands upfront. These help to inspect or learn more about a unit.\nReceive help: systemctl help ssh.service Show the unit file: systemctl cat ssh.service Show all unit settings: systemctl show ssh.service Unit types .service This is a very common unit and refers to a service that runs in the background. Typically these are daemons that perform the whole type the system is running. Another option is a script that is executed by a trigger, like a repeating task.\n.timer Another common unit is the timer. It\u0026rsquo;s a scheduler, similar to cronjobs. Usually a timer and service file are paired together. The timer does the scheduling and then triggers the service.\nsystemctl list-timers\n.target A state within the service manager. It groups services and performs synchronization during the boot process or when the state changes. A good example of this is services that require the network stack to be up and running, like nginx. The network target can only be started when the basic functionality of the system is active, like some devices. In other words, it is a chain of events.\nTo see the time-critical tasks during the boot, we can use the systemd-analyze tool.\n# systemd-analyze critical-chain The time when unit became active or started is printed after the \u0026#34;@\u0026#34; character. The time the unit took to start is printed after the \u0026#34;+\u0026#34; character. graphical.target @19.498s └─multi-user.target @19.498s └─snapd.seeded.service @17.537s +1.874s └─basic.target @17.467s └─sockets.target @17.466s └─snapd.socket @17.444s +15ms └─sysinit.target @17.419s └─cloud-init.service @16.952s +465ms └─systemd-networkd-wait-online.service @3.048s +13.900s └─systemd-networkd.service @3.010s +36ms └─network-pre.target @3.009s └─cloud-init-local.service @2.383s +625ms └─systemd-remount-fs.service @445ms +62ms └─systemd-journald.socket @380ms └─-.mount @345ms └─-.slice @345ms If we want to see how specific targets are linked, we can also use systemctl list-dependencies that shows the relationship between different units.\n# systemctl list-dependencies local-fs.target local-fs.target ● ├─-.mount ● ├─boot.mount ○ ├─systemd-fsck-root.service ● └─systemd-remount-fs.service To have the local-fs.target up and running, a set of mount and file system checks have to be performed. We can see that the \u0026lsquo;fsck\u0026rsquo; service is not running, as that only gets triggered when it is actually needed. The other three units (two mount units, one service) are active.\n.socket A socket description for socket based activation. This is typically something like a network socket, file system FIFO file, or other forms of inter-process communication (IPC).\nsystemctl list-sockets\n.device Devices that demand systemd management, are stored in this unit type, like the initrd (ramdisk).\n.mount Defines a mount point on the system\n.automount Defines a mount point that will be mounted automatically.\nsystemctl list-automounts\n.path Defines a file or directory to monitor and take trigger a related service. This is for example great to monitor for changes and then initiate a related action.\nsystemctl list-paths\n.swap Define the swap space, which is used when there is not enough memory.\n.snapshot Dynamic snapshot of the systemd runtime state. This is not a file type like the others, but an auto-generated definition.\n.slice Defines cgroups, which are the Linux Control Groups. They restrict access to resources, such as memory and CPU.\n.scope These are created automatically and are used to manage sets of systemd processes, such as a user session.\n","permalink":"https://linux-audit.com/systemd/systemd-units-and-their-purpose/","tags":["cgroups","Linux","mount","swap","systemd"],"title":"Systemd units and their purpose"},{"categories":["System Administration"],"contents":"Many Linux distributions are using systemd as its system and service manager, so it makes sense to get to know how to use it properly. In this cheat sheet we collected the commands that everyone should know to get the most out of it.\nCommonly used systemctl commands and options Command Action daemon-reload Reload the manager after making changes to a systemd unit file disable UNIT Disable the unit edit UNIT Create an override for a unit edit --full UNIT Edit the main unit file, usually better to create override enable UNIT Enable the unit is-enabled UNIT Check if the unit is enabled list-units Show active units list-units --all Show all units including the ones that are inactive list-unit-files List unit files and their state mask UNIT Fully disable the unit (manual start not possible) reload UNIT Request unit to reload configuration (not always possible, depends on support) restart UNIT Restart the unit start UNIT Start the unit status UNIT Show the status details of the unit stop UNIT Stop the unit unmask UNIT Reactivate unit so it can be started again Options Option Action --legend=false Don\u0026rsquo;t show header or number of matching units --state=failed Filter unit by state (failed) Show all units The aptly named subcommand list-units shows an overview of available units. It groups them by type (automount, device, scope, path, mount, service, timer, etc).\n# systemctl list-units UNIT LOAD ACTIVE SUB DESCRIPTION mnt-websites.automount loaded active running NFS automount for /mnt/websites proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary Executable File Formats... dev-loop1.device loaded activating tentative /dev/loop1 dev-loop6.device loaded activating tentative /dev/loop6 Another useful way to show the available units, is with list-unit-files. The output is brief, but includes the name and the state.\n# systemctl list-unit-files UNIT FILE STATE VENDOR PRESET mnt-websites.automount enabled enabled proc-sys-fs-binfmt_misc.automount static - -.mount generated - boot.mount generated - dev-hugepages.mount static - dev-mqueue.mount static - mnt-websites.mount enabled enabled proc-fs-nfsd.mount static - proc-sys-fs-binfmt_misc.mount disabled disabled run-rpc_pipefs.mount generated - snap-core20-2182.mount enabled enabled snap-core20-2264.mount enabled enabled snap-lxd-27948.mount enabled enabled snap-lxd-28373.mount enabled enabled snap-snapd-21184.mount enabled enabled snap-snapd-21465.mount enabled enabled sys-fs-fuse-connections.mount static - sys-kernel-config.mount static - sys-kernel-debug.mount static - sys-kernel-tracing.mount static - var-lib-nfs-rpc_pipefs.mount static - Inspect single systemd unit Useful commands for inspecting systemd units include:\ncat list-dependencies show status Obtain basic details The basic command for inspecting a unit, is with the status command.\n# systemctl status ssh.service ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; preset: enabled) Active: active (running) since Tue 2024-03-12 10:14:53 UTC; 1 month 13 days ago Docs: man:sshd(8) man:sshd_config(5) Main PID: 446 (sshd) Tasks: 1 (limit: 4691) Memory: 28.2M CPU: 35min 39.234s CGroup: /system.slice/ssh.service └─446 \u0026#34;sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\u0026#34; Apr 24 09:04:28 example.com sshd[322491]: pam_env(sshd:session): deprecated reading of user environment enabled Apr 24 14:59:57 example.com sshd[323298]: Accepted publickey for username from 1.2.3.4 port 39518 ssh2: ED25519 SHA256:SK0sf09a8s09df89a0sdfLUYS2sSkjKJyYOISDFJKLS Apr 24 14:59:57 example.com sshd[323298]: pam_unix(sshd:session): session opened for user username(uid=10000) by (uid=0) Apr 24 14:59:57 example.com sshd[323298]: pam_env(sshd:session): deprecated reading of user environment enabled Apr 24 17:24:34 example.com sshd[323424]: Accepted publickey for username from 1.2.3.4 port 58572 ssh2: ED25519 SHA256:SK0sf09a8s09df89a0sdfLUYS2sSkjKJyYOISDFJKLS Apr 24 17:24:34 example.com sshd[323424]: pam_unix(sshd:session): session opened for user username(uid=10000) by (uid=0) Apr 24 17:24:34 example.com sshd[323424]: pam_env(sshd:session): deprecated reading of user environment enabled Apr 24 17:53:27 example.com sshd[323504]: Accepted publickey for username from 1.2.3.4 port 47364 ssh2: ED25519 SHA256:SK0sf09a8s09df89a0sdfLUYS2sSkjKJyYOISDFJKLS Apr 24 17:53:27 example.com sshd[323504]: pam_unix(sshd:session): session opened for user username(uid=10000) by (uid=0) Apr 24 17:53:27 example.com sshd[323504]: pam_env(sshd:session): deprecated reading of user environment enabled The status command will reveal a lot of basic details, such as its state, documentation, process ID, memory and CPU usage, and the last 10 lines of related logging.\nShow the unit the file and defined settings To see the configuration of the unit, use the cat subcommand.\nsystemctl cat ssh.service\nThis is what a typical service unit looks like:\n# /lib/systemd/system/ssh.service [Unit] Description=OpenBSD Secure Shell server Documentation=man:sshd(8) man:sshd_config(5) After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] EnvironmentFile=-/etc/default/ssh ExecStartPre=/usr/sbin/sshd -t ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/usr/sbin/sshd -t ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartPreventExitStatus=255 Type=notify RuntimeDirectory=sshd RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target Alias=sshd.service The first line shows where the file is located on the disk, it is not an actual comment line\nThe options defined in the \u0026lsquo;[Service]\u0026rsquo; section are just a few. To see all related settings and restrictions of the service, use the show subcommand. This is a great way to audit all services and see where additional hardening can be applied.\nsystemctl show ssh.service\nExample output:\nType=notify ExitType=main Restart=on-failure NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s TimeoutStartFailureMode=terminate TimeoutStopFailureMode=terminate RuntimeMaxUSec=infinity RuntimeRandomizedExtraUSec=0 WatchdogUSec=0 WatchdogTimestampMonotonic=0 List dependencies To show the related dependencies of a particular service, use the aptly named list-dependencies command and specify the unit.\n# systemctl list-dependencies ssh.service ssh.service ● ├─-.mount ● ├─system.slice ● └─sysinit.target ● ├─apparmor.service ● ├─dev-hugepages.mount ● ├─dev-mqueue.mount ● ├─kmod-static-nodes.service ● ├─proc-sys-fs-binfmt_misc.automount ● ├─sys-fs-fuse-connections.mount ● ├─sys-kernel-config.mount ● ├─sys-kernel-debug.mount ● ├─sys-kernel-tracing.mount ● ├─systemd-ask-password-console.path ● ├─systemd-binfmt.service ○ ├─systemd-firstboot.service ● ├─systemd-journal-flush.service ● ├─systemd-journald.service ○ ├─systemd-machine-id-commit.service ● ├─systemd-modules-load.service ● ├─systemd-network-generator.service ○ ├─systemd-pcrphase-sysinit.service ○ ├─systemd-pcrphase.service ○ ├─systemd-pstore.service ● ├─systemd-random-seed.service ○ ├─systemd-repart.service ● ├─systemd-resolved.service ● ├─systemd-sysctl.service ● ├─systemd-sysusers.service ● ├─systemd-timesyncd.service ● ├─systemd-tmpfiles-setup-dev.service ● ├─systemd-tmpfiles-setup.service ● ├─systemd-udev-trigger.service ● ├─systemd-udevd.service ● ├─systemd-update-utmp.service ● ├─cryptsetup.target ● ├─integritysetup.target ● ├─local-fs.target ● │ ├─-.mount ● │ ├─boot-efi.mount ○ │ ├─systemd-fsck-root.service ● │ └─systemd-remount-fs.service ● ├─swap.target ● └─veritysetup.target Disable or enable systemd units Units can be enabled or disabled, to let the systemd service manager know if a unit should be started during the boot cycle of the operating system. Not all services are started at boot, as some are dependencies or triggered by a timer.\nWhen a service is not enabled yet, this can be done using the enable command. The disable command does the opposite. When using the --now option together with enable, we tell systemctl to enable the service and also start it. This is a shortened version of enable and start in two separate commands.\nsystemctl enable --now UNIT\nTo disable a service so that it can\u0026rsquo;t be started anymore, the disable command is not sufficient. Instead, we need to mask it. This command tells the service manager that we want to block execution. This is useful for software that is not configured yet, or components that are not required. To enable a \u0026lsquo;masked\u0026rsquo; unit again, use the unmask command.\nDealing with failed units Systemd units can get into a failed state, for example if execution of the related program gave an unexpected exit code. To show all failed systemd units, filter them by state.\nsystemctl --state=failed\nThe option --failed is shorter option and works as well\nUnits can get into a failed state due to settings, missing dependencies, external conditions, issues with permissions and so. The troubleshooting of failed systemd units may provide hints. After resolving an issue, try a restart or reset of the unit.\nWe can reset all failed units. Optionally, the unit name can be specified to target only that unit.\nsystemctl reset-failed\nTimers Timers are the replacement of cronjobs. Use the list-timers command to show them.\n# systemctl list-timers NEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2024-04-25 11:14:19 UTC 2h 10min left Wed 2024-04-24 23:20:56 UTC 9h ago motd-news.timer motd-news.service Thu 2024-04-25 14:28:29 UTC 5h 25min left Thu 2024-04-25 03:55:34 UTC 5h 7min ago apt-daily.timer apt-daily.service Thu 2024-04-25 19:15:01 UTC 10h left Wed 2024-04-24 19:15:01 UTC 13h ago update-notifier-download.timer update-notifier-download.service Thu 2024-04-25 19:26:06 UTC 10h left Wed 2024-04-24 19:26:06 UTC 13h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Fri 2024-04-26 00:00:00 UTC 14h left Thu 2024-04-25 00:00:02 UTC 9h ago dpkg-db-backup.timer dpkg-db-backup.service Fri 2024-04-26 00:00:00 UTC 14h left Thu 2024-04-25 00:00:02 UTC 9h ago logrotate.timer logrotate.service Fri 2024-04-26 00:40:00 UTC 15h left Thu 2024-04-25 08:43:29 UTC 19min ago fwupd-refresh.timer fwupd-refresh.service Fri 2024-04-26 03:06:36 UTC 18h left Thu 2024-04-25 04:15:34 UTC 4h 47min ago man-db.timer man-db.service Fri 2024-04-26 06:44:11 UTC 21h left Thu 2024-04-25 06:32:31 UTC 2h 30min ago apt-daily-upgrade.timer apt-daily-upgrade.service Sun 2024-04-28 03:10:21 UTC 2 days left Sun 2024-04-21 03:10:56 UTC 4 days ago e2scrub_all.timer e2scrub_all.service Mon 2024-04-29 01:12:43 UTC 3 days left Mon 2024-04-22 01:14:34 UTC 3 days ago fstrim.timer fstrim.service Fri 2024-05-03 23:10:43 UTC 1 week 1 day left Tue 2024-04-23 12:33:56 UTC 1 day 20h ago update-notifier-motd.timer update-notifier-motd.service The command will reveal when the last execution of the timer was and the upcoming one, including the time left. Also, the related service file will be displayed, making it easier to see the relation between a timer and another unit.\nSystem state As systemd is a system and service manager, it can also respond to commands to stop or reboot the system.\nsystemctl reboot systemctl poweroff systemctl suspend systemctl hibernate ","permalink":"https://linux-audit.com/cheat-sheets/systemctl/","tags":["cheatsheet","howto","linux","systemctl","systemd"],"title":"systemctl cheat sheet"},{"categories":["System Administration"],"contents":"Common journalctl options Long option Short option What the option does --catalog -x Show log lines with additional help or suggestions where available --disk-usage Show size of the archived and active journals --follow -f Track changes, like tail -f --lines= -n Show X number of lines (most recent) --output= -o Define what output format should be used for journal entries --pager-end -e Go to the end of the pager output, so the last entries are visibible --reverse -r Reverse output, newest on top --since= -S Limit the data to a specific period (begin) --unit -u Specify the unit when querying the logs or taking an action --until= -U Limit the data to a specific period (end) --vacuum-files Trim journal logs by number --vacuum-size Clear log entries from journal logs by specifying total size --vacuum-time Clear log entries from journal logs by specifying time (age) --verify Integrity check of the journals Showing basic details of the journals The size of the journals can displayed with --disk-usage, which may be useful when the file system is getting full.\n# journalctl --disk-usage Archived and active journals take up 160.0M in the file system. To verify the integrity of the journals, use the --verify option.\n# journalctl --verify PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/system@ca889eb2eae24e41b37a50d33bad131c-0000000000000001-00060ed90326924f.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000036b8-000611a51acdc7d0.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/user-1000@aeb5e2f412954ecfaa870c245338cb93-00000000000004e2-00060ed9041f690a.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/user-1000.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/system@ca889eb2eae24e41b37a50d33bad131c-0000000000014c26-000613a20f360102.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/system.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/user-1000@aeb5e2f412954ecfaa870c245338cb93-0000000000014e03-000613d7a3bb17e3.journal PASS: /var/log/journal/d8bd6473290d43a9942eaba0a506a454/system@ca889eb2eae24e41b37a50d33bad131c-0000000000003334-0006113d51794916.journal Querying the journals There are many ways to query the journals. One of them is simply running journalctl and start scrolling. But there are better ways!\nQuery by time or period Show messages of today with the --since= option and define the period.\njournalctl --since=\u0026quot;today\u0026quot;\nTo shorten the period, we can tell it to show only very recent entries of fifteen minutes ago or newer.\njournalctl --since=\u0026quot;15 minutes ago\u0026quot;\nWe can also provide a range, with the combination of since and until.\njournalctl --since=\u0026quot;2024-02-01\u0026quot; --until=\u0026quot;2024-04-01\u0026quot;\nFor troubleshooting it may help to increase the period, but include a unit name to strip out much of the unneeded entries.\njournalctl --unit ssh.service --since=\u0026quot;1 week ago\u0026quot;\nQuery by priority or facility Journalctl allows to query by priority. Here are the available levels:\nPriority level Name 0 emerg 1 alert 2 crit 3 err 4 warning 5 notice 6 debug 7 info Use the short notation, but now include the unit name in the output, and limit messages to a priority (including the ones with lower number, meaning a higher priority).\njournalctl -S \u0026quot;today\u0026quot; --output=with-unit --priority=err\nOnly show some levels (notice, debug, and info)\njournalctl -p 5..7\nWhen querying by facility, which are common with syslog, define the right value. To know the available facilities, use the \u0026lsquo;help\u0026rsquo;.\n# journalctl --facility=help Available facilities: kern user mail daemon auth syslog lpr news uucp cron authpriv ftp 12 13 14 15 local0 local1 local2 local3 local4 local5 local6 local7 Query by string Similar to the grep tool, there are a few options available to search in the journals. It shares the same name, but is an option instead.\njournalctl --grep \u0026quot;[bB]lock\u0026quot;\nRegular expressions are allowed, so be aware of case-sensitive filtering.\nWant to search and not worry about lowercase and uppercase?\njournalctl --case-sensitive=false --grep \u0026quot;block\u0026quot;\nQuery only by priority ERROR Show only the entries flagged with priority ERROR.\njournalctl -p err\nOr since last boot:\njournalctl -b -p err\nLimit output and follow Journalctl allows to limit the output to a specific number of lines. To show the last 10 lines, which is equal to --lines=10, we can use -n.\njournalctl -n\nWe can also combine it with a unit, and show only 5 lines.\njournalctl -u ssh.service -n 5\nKeep track of new additions, we can use the --follow option, similar to tail -f.\njournalctl --follow\nCleaning up the journals When the journal gets too big, decrease its size by performing a vacuum action.\njournalctl --vacuum-size=256M\nIt is also possible to set a time period instead, like 4 weeks.\njournalctl --vacuum-time=4w\nAnother possibility is defining the number of logs.\njournalctl --vacuum-files=5\nOther useful options? Did I miss something that really should be included in this cheat sheet? Let it know!\n","permalink":"https://linux-audit.com/cheat-sheets/journalctl/","tags":["cheatsheet","howto","journalctl","syslog","systemd"],"title":"journalctl cheat sheet"},{"categories":["Nginx","System Administration","Web"],"contents":"Nginx makes it very easy to define headers to improve the caching of static resources such as images, style sheets and JavaScript files. To accomplish this task, we only need a few things to configure.\nMIME types configuration First we have to ensure that the configuration of the MIME types are correct. MIME is short for Multipurpose Internet Mail Extensions and defines what a specific file contains. For example, a file with the file extension .png is most likely a PNG file. Or in MIME language, a image/png. The first part is the category (an image), the second one the specific type (PNG). An HTML file is text and is mapped as text/html.\nThe definitions of the MIME types are usually stores in a file mime.types. The main configuration file of nginx, which is typically /etc/nginx/nginx.conf, usually already includes it by default.\nhttp { include /etc/nginx/mime.types; } Check your configuration if this is the case. Also check if it has a set of defined MIME types.\nDefine a map The second step is to set a map and use the sent_http_content_type variable. This contains the value that the server will return to the client, based on the definition it found in the MIME types.\nmap $sent_http_content_type $example_com_expire_time { application/javascript 1d; application/json 1d; application/pdf 90d; font/woff 90d; font/woff2 90d; ~image/ 7d; image/jpeg 30d; image/png 30d; image/svg+xml 30d; image/webp 30d; image/x-icon 30d; text/css 1d; text/html 1h; text/xml 4h; # atom.xml default 4h; } Tip: if you have multiple virtual hosts configured, include the name of your virtual host in the variable that ends with expire_time. This way configuration settings won\u0026rsquo;t conflict with each other, resulting in unexpected caching values.\nIn the map above we defined several common MIME types. The default line tells what the default value is, when the MIME type does not match any of the defined ones. In this case we set the value to four hours, but by setting is to \u0026lsquo;off\u0026rsquo;, you can disable caching for unspecified types.\nConfigure your virtual host With the map in place, it is time to enable the setting. This is usually done in your server definition, the place where the virtual host is configured. This way all returned files will receive the related header.\nhttp { server { server_name example.com; ### Caching expires $example_com_expire_time; location / { # ... configuration } } } Restart nginx and test After saving the changes, perform a configuration test first.\nnginx -t Now we can use a tool like curl to test if our headers are set correctly.\ncurl --head https://example.com\nIf all is well, you most likely will see the following two headers:\nExpires Cache-Control The Expires will show you the actual date and time, where the Cache-Control defines the time in seconds (with max-age).\nBonus tip: automated testing To easily test if all your file types are correct, consider using a configuration file that can be used together with curl. Then insert multiple types of requests on your website, such as the home page, the path to the logo, your sitemap, your robots.txt, etc. This way you can easily repeat the test and see if multiple MIME types are properly detected and get the right caching instructions. See the cheat sheet for all tips regarding using curl.\n\u0026raquo; Mastering the tool: curl cheat sheet\ncurl cheat sheet Conclusion Setting headers to allow clients to cache static content is fairly easy to do. Define the right map, insert the header, and finally test if everything is working as expected.\nDid you already secure your nginx configuration? Have a look at the nginx security hardening guide.\n","permalink":"https://linux-audit.com/web/nginx-adding-expires-header-to-improve-caching/","tags":["howto","linux","nginx","web server"],"title":"Adding the Expires header to improve caching static content in nginx"},{"categories":["Cheat Sheets","System Administration","Web"],"contents":"One of the best HTTP clients is the open source tool curl. With ongoing development and continuously new updates, it is worth getting everything out of this powerful tool!\nBasic options Some of the curl options are used a lot in combination with others. So it is good to know these if you are a beginner or used curl before.\nLong option Short option What the option does --compressed Reduce data transfer size by sending Accept-Encoding header (server support needed) --fail -f Be silent when a failure occurs, useful for scripts that have some failure handling --head -I Show the response headers, not the normal output (HTML/file) --include -i Include the response headers as well (the part that --head shows) --output file.txt -o file.txt Write the output to a file instead of screen --request POST -X POST Perform a POST request instead of the default GET --show-error -S Show only errors when used with --silent --silent -s Be quiet with output --verbose -v Verbose output, great for debugging Creating a shell script? Use the long format option, as this improves the readability. For quick use of curl on the command-line consider using the short notation of the related option.\nHTTP protocol version protocols Option Usage --http1.0 Use HTTP/1.0 protocol --http1.1 Use HTTP/1.1 protocol --http2 Try using the HTTP/2.0 protocol (depends on HTTP or HTTPS) --http3 Try using the HTTP/3 protocol Enable data encoding (data compression) By default, curl does no data encoding or decoding. To allow receiving compressed data, used the --compressed option, or define the Accept-Encoding header\ncurl --compressed --head https://example.com/\nWant to test just a specific type of compression, define the header and tell what you support. Multiple options are possible using a comma as separator.\ncurl --head --header 'Accept-Encoding: br,gzip' https://example.com/\nTLS and Certificates protocols With curl being able to do many things, it definitely knows how to deal with all kinds of protocols. Some interesting options in this area:\nOption Short option Usage --cert-status Check the status of certificate using OCSP --insecure -k Skip verification of certificate, useful when having a self-signed certificate --tls-max Set maximum TLS version --tlsv1.1 Set minimum version TLSv1.1 --tlsv1.2 Set minimum version TLSv1.2 --tlsv1.3 Set minimum version TLSv1.3 Test SSL verification The SSL verification steps within curl can show its result using the ssl_verify_result variable.\ncurl --compressed --head --write-out %{ssl_verify_result} https://example.com\nA value of \u0026lsquo;0\u0026rsquo; means the verification went as expected.\nImportant: If you use this in a script, consider also checking the exit state of the curl command itself first, because it may show a \u0026lsquo;0\u0026rsquo; while something like DNS resolution or the connection did went well in the first place.\nTest SSL certificate status using OCSP Perform the request and include the certification status check:\ncurl --cert-status --head --verbose https://example.com/\nCurl will show a single line in the output to indicate the outcome of the check.\n* SSL certificate status: good (0) Show expiry date of a SSL certificate Curl can be used to display when a SSL certificate will expire. By extracting the right line from the verbose output, we can get a single line with the information.\ncurl --verbose --head 2\u0026gt;\u0026amp;1 https://example.com/ | grep '\\* expire date:'\nThis information is great for showing to a human, but not for automated processing. If we extract the specific fields and replace the month name with a two-digit number, then it may result in a more common format for storage.\ncurl --verbose --head 2\u0026gt;\u0026amp;1 https://example.com/ | \\ grep \u0026#39;\\* expire date:\u0026#39; | \\ sed \u0026#39;s/\\* expire date: //\u0026#39; | \\ awk \u0026#39;{m=sprintf(\u0026#34;%02d\u0026#34;, (index(\u0026#34;JanFebMarAprMayJunJulAugSepOctNovDec\u0026#34;,$1)+2)/3)};END{print $4\u0026#34;-\u0026#34;m\u0026#34;-\u0026#34;$2}\u0026#39; Important note: This one-liner has no checks for failures such a connection errors. If you want to use this in production, consider rewriting it into a full script with proper error checks. Or use a dedicated tool for SSL certification expiry monitoring. That a tool can be used in a wide range of situations does not mean it is the best option.\nTesting specific TLS versions For security testing and troubleshooting it may be useful to test against different protocol versions. Especially if nginx configuration hardening has been applied to limit the allowable TLS versions.\nIf you only enable TLSv1.2 and later, we can test the minimum TLS version 1.0 en 1.1 as its max. This request then should fail:\ncurl --head --tls-max 1.1 --tlsv1.0 --verbose https://example.com/\nIf we want to set the minimum version to TLS version 1.2, it should succeed if TLSv1.2 and TLSv1.3 are allowed:\ncurl --head --tlsv1.2 --verbose https://example.com/\nFiles and downloads The --output option can be used to write the output of a request to a file instead of the screen (stdout).\ncurl --output myfilename https://example.com/randomfilename.txt\nUse --remote-name to save the file with the same file name as the server.\ncurl --remote-name https://example.com/numbers.txt\nIn this example, the local file to store will be named numbers.txt.\nHeaders headers In need to change a request header? Curl makes this possible with the --header option, followed by a quoted string.\ncurl --header \u0026quot;Secret: true\u0026quot; https://example.com/\nSpecify the User-Agent string Sometimes a download might be blocked due to server blocking curl or wget as its User-Agent string. Use the --user-agent option to define another value.\ncurl --user-agent \u0026quot;I-am-not-Firefox\u0026quot; https://example.com/\nAutomated testing Curl can be used manually, but is also a great tool for automating tests and displaying or storing the result.\nShow response headers headers We can use curl to test our website and see if the right headers are returned. For example if the content-type or content-encoding is set correctly, or something like the headers related to caching.\nTo quickly test the output, use a HEAD request (--head) in combination with the --no-progress-meter option. This last one prevents curl showing a progress output.\ncurl --head --no-progress-meter https://example.com/ | \\ grep \u0026#39;^content-type\\|^date\\|^expires\\|^content-encoding\u0026#39; Only return the HTTP status code headers With the right combination of options, we can curl to show only the HTTP status code. Great for building a small link checker or to perform a monitoring task.\ncurl --compressed --head --output /dev/null --silent --write-out '%{http_code}' https://example.com/\nWhy this combination?\n--compressed: when possible limit data traffic (optional, but typically a good idea) --head: don\u0026rsquo;t query full page, just the response headers --output \u0026gt;/dev/null: don\u0026rsquo;t show or store the output of the headers --silent: do not show any errors --write-out \u0026lsquo;%{http_code}\u0026rsquo;: only show the HTTP status code Use a configuration file for repeated tasks automation Want to test the same set of URLs, consider creating a custom configuration file.\ncurl --config myconfig.conf\nExample configuration:\n# Test main page headers, with output to screen url = \u0026#34;https://example.com/\u0026#34; --compressed --connect-timeout 3 --head --max-time 5 --no-progress-meter user-agent = \u0026#34;michael-test/1.0\u0026#34; # Test logo, send output to file, include a referrer url = \u0026#34;https://example/logo.svg\u0026#34; --compressed --connect-timeout 3 --head --max-time 5 --no-progress-meter output = \u0026#34;output-logo.txt\u0026#34; referer = \u0026#34;https://example.com/\u0026#34; Note: in the HTTP spec there is a typo, so that is why it is misspelled above as well.\nMeasure web server performance headers automation Curl has a great set of variables available to display. That is nice, but it would be better if we could combine those into a neat formatted output.\nThe first step is to create a small shell script to do the query.\n#!/bin/sh set -o nounset if [ $# -eq 0 ]; then echo \u0026#34;Error: URL required to query\u0026#34;; exit 1; fi # tr-encoding requests a compressed resource, however it is better to define what we can accept. -H \u0026#34;Accept-Encoding: br,gzip,deflate\u0026#34; curl --silent --compressed --output /dev/null --write-out \u0026#34;@measure-webserver-performance.format\u0026#34; \u0026#34;$1\u0026#34; # EOF Next step is to create a formatted template with the variables that you want to show.\ntime_namelookup: %{time_namelookup}\\n time_connect: %{time_connect}\\n time_appconnect: %{time_appconnect}\\n time_pretransfer: %{time_pretransfer}\\n time_redirect: %{time_redirect}\\n time_starttransfer: %{time_starttransfer}\\n http_code: %{http_code}\\n http_version: %{http_version}\\n scheme: %{scheme}\\n remote_ip: %{remote_ip}\\n content_type: %{content_type}\\n num_redirects: %{num_redirects}\\n ssl_verify_result: %{ssl_verify_result}\\n Total bytes downloaded: %{size_download}\\n Total bytes of the downloaded headers: %{size_header}\\n Total bytes sent in the HTTP request: %{size_request}\\n Total bytes that were uploaded: %{size_upload}\\n ----------\\n time_total: %{time_total}\\n When we run the script together with an URL, we get a nice piece of output.\n# ./measure-webserver-performance https://linux-audit.com time_namelookup: 0.002094 time_connect: 0.013645 time_appconnect: 0.055099 time_pretransfer: 0.055141 time_redirect: 0.000000 time_starttransfer: 0.070597 http_code: 200 http_version: 2 scheme: HTTPS remote_ip: 89.41.171.41 content_type: text/html num_redirects: 0 ssl_verify_result: 0 Total bytes downloaded: 5888 Total bytes of the downloaded headers: 1322 Total bytes sent in the HTTP request: 119 Total bytes that were uploaded: 0 ---------- time_total: 0.070788 Using timeouts connection When doing automated testing, it makes sense to limit the amount of time a full request may take. Use the --max-time option for that.\ncurl --max-time 10 https://example.com/\nTo set the maximum time that the connection phase may take, use the --connect-time. An example to combine them.\ncurl --connect-timeout 3 --max-time 10 https://example.com/\nSend POST with JSON data Besides the usual HEAD and GET requests, curl can also post data. In this case it needs to be defined using the --request option. The actual data is provided by using the --data option.\ncurl --header \u0026quot;Content-Type: application/json\u0026quot; --request POST --data '{\u0026quot;name\u0026quot;:\u0026quot;michael\u0026quot;,\u0026quot;value\u0026quot;:\u0026quot;123\u0026quot;}' https://example.com/api/\nAnother option is to send so-called form-encoded data. This is also a POST request and has the type \u0026lsquo;application/x-www-form-urlencoded\u0026rsquo;.\ncurl --data 'name=michael' http://example.com/my-form/\nTip: use --data @myfilename to retrieve the data from a file, or use @\u0026rsquo;-\u0026rsquo; to read from STDIN.\nCrafting special requests security Sometimes curl is too smart for its own good. To avoid it normalizing the requested URI, such as /test/../file.txt into /file.txt, there is the --path-as-is option.\ncurl --path-as-is https://example.com/../../etc/passwd\nThis is useful for security professionals to test against some file inclusion weaknesses or trying a path traversal attack. For webmasters it is a great option to see if your block filters are working correctly.\nOther useful options? Did I miss something that really should be included in this cheat sheet? Let it know!\n","permalink":"https://linux-audit.com/cheat-sheets/curl/","tags":["curl","cheatsheet","howto","one-liner"],"title":"curl cheat sheet"},{"categories":["Nginx","Web"],"contents":"Learn how to secure your nginx configuration with this hardening guide. It includes examples and tips to implement security measures step by step.\nWhy harden your nginx configuration? Nginx is known for its speed and modular support. It even has multiple security safeguards to prevent or limit common attacks. That\u0026rsquo;s a great start, but not enough for a production system with sensitive information stored on it. Even if you don\u0026rsquo;t host sensitive information, having it hacked is no fun, right?\nIn this guide we go step by step and secure the nginx configuration. Each time a small security measure is implemented, making the hosted websites a bit more secure.\nWarnings and tips Before we start making changes to the system, it is a good idea to have a good backup. Each step might break websites, so take it easy with the deployment and monitor your log files.\nMake a backup of the nginx configuration files Ensure that access logging is enabled (should not be disabled) Some configuration parameters need to be made in the http definition, while others are placed in server or location definition Before restarting, use nginx -t to test the new configuration For some changes a restart of nginx is needed, while for most a reload is sufficient As every system is different, have a good look at your situation before making changes Consider the type of clients that need to connect to your web server, such as desktop/mobile and their support for newer technologies Apply hardening to your full application stack (OS, firewall, backend applications), as nginx is just one part of it Generic changes Test your configuration Before we make any changes, let\u0026rsquo;s start with testing the configuration.\nnginx -t\nWant to see the full configuration, including a test?\nnginx -T\nIf no warnings or errors show up, we can continue!\nDisable nginx version number information leakage Typically it is better to reveal as less as possible when it comes to running software components. A good starter with nginx is to disable the version. This is done in the http definition within the configuration. This section is usually part of the /etc/nginx/nginx.conf file.\nhttp { server_tokens off; # Other HTTP configurations options } HTTPS configuration Most websites run nowadays on HTTPS. There is almost no reason to run just on HTTP, especially with SSL certificates being available for free.\nBasic SSL configuration data encryption The first thing to configure is the SSL certificate and the related key. These need to be obtained from your Certificate Authority (CA). This can be from your own organization or an external one like Let\u0026rsquo;s Encrypt.\nhttp { server { listen 443 ssl; server_name example.com; ssl_certificate /path/to/example.com.crt; ssl_certificate_key /path/to/example.com.key; # Other SSL settings (TLS versions, cipher suites) } } Enable OCSP performance Enable OCSP to improve the performance of the TLS connection.\nRationale to enable OCSP OCSP is the abbreviation of Online Certificate Status Protocol. It checks the validity status of a certificate in real-time, so a client does not have to use a revocation list. OCSP stapling improves the performance of the validation checks by using a signed and time-stamped version of the OCSP response. This is stored on the web server and refreshed on a regular schedule. The result of this verification is provided during the handshake and reduces another validation step on the system of the user.\nhttp { server { # OCSP ssl_trusted_certificate /path/to/example.com.crt; ssl_stapling on; ssl_stapling_verify on; resolver 1.1.1.1 1.0.0.1 [2606:4700:4700::1111] [2606:4700:4700::1001]; } } Replace the resolver with name servers that you trust and have a good performance\nTLS versions data encryption performance Disable older TLS versions.\nRationale to limit TLS protocols Implement modern TLS protocol versions and disable those with known issues. Not sure what protocols are currently used? Log them using the $ssl_protocol and $ssl_cipher variables via the log_format function.\nhttp { server { ssl_protocols TLSv1.2 TLSv1.3; } } Cipher suites data encryption Use strong ciphers and consider performance.\nRationale to define cipher sets A cipher suite is a set of algorithms. It helps to secure the network connection and uses the TLS protocol within the nginx configuration. When selecting the right set of ciphers, one has to look at ciphers that are considered to be secure, but also have a good performance.\nhttp { server { ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305; ssl_prefer_server_ciphers off; } } This set of ciphers comes from Mozilla and has a good compatibility with clients. If you have a specific virtual host that only requires very modern clients to connect, consider disabling TLSv1.2 (earlier section) and use a more restricted set of ciphers.\nIn the past, many guides suggested to let the server decide what cipher to use. But when it comes to performance, typically the client can make a better decision. For that reason the ssl_prefer_server_ciphers is set to off.\nUseful links:\nMozilla SSL configuration generator Enable Kernel TLS offload performance Consider performance for a better user experience.\nRationale to enable KTLS KTLS is a method to offload the handling of TLS operations in the kernel itself, instead of a user process like nginx. This may increase the performance of the TLS processes, such as handshake, data encryption, etc.\nRequirements:\nLinux \u0026gt;= 4.13 NGINX \u0026gt;= 1.21.4 OpenSSL \u0026gt;= 3.0.0 http { sendfile on; server { # Enable kernel TLS ssl_conf_command Options KTLS; } } Restrict access authentication limit access Limit access to resources by defining which systems or users can access them.\nUsing IP address When some resources can be easily filtered by an IP address, use a combination of allow with the deny all.\nhttp { server { location /mystatus { stub_status on; allow 1.2.3.4; deny all; break; } } } Using basic authentication authentication limit access When multiple users (with different IP addresses) need to access a specific location, basic authentication could be used. This is definitely not the most secure type of authentication and access control, but for some endpoints it could be an additional security measure to restrict access. If there is an application running behind the endpoint, then use that to arrange authentication and authorization.\nhttp { server { location /secret { auth_basic \u0026#34;Restricted Content\u0026#34;; auth_basic_user_file /etc/nginx/.htpasswd; # Other location directives } } } The .htpasswd file can be provisioned with a combination of usernames and password using the htpasswd command.\nRate limiting system resources To prevent aggressive clients, implement rate limiting. Typically this is done at the http definition in combination with the server or a particular location. It is also possible to define multiple limit rates, depending on the specific paths and how many HTTP requests are common.\nhttp { limit_req_zone $binary_remote_addr zone=globalratelimit:10m rate=10r/s; server { location / { limit_req zone=globalratelimit burst=20; # Other location directives } } } Protocols and methods Disable older protocols limit access system resources Most clients use currently the HTTP/1.1 or HTTP/2.0 protocol to connect to a web server. The older HTTP/1.0 is really old and lacks a wide range of features:\nlack of data compression limited set of HTTP error codes multiple requests per connection If you don\u0026rsquo;t need to support these very old clients, then you could consider blocking them and reduce data traffic.\nhttp { server { location / { # No longer accept HTTP1.0 requests, show 426 (Upgrade Required), needs to be in location block as we are adding headers. if ($server_protocol = HTTP/1.0) { add_header Upgrade \u0026#34;HTTP/2.0\u0026#34;; add_header Connection \u0026#34;Upgrade\u0026#34;; return 426 \u0026#34;Upgrade Required: Upgrade your client to support more modern HTTP protocol\u0026#34;; break; } } } } Define allowed HTTP request methods limit access security When we host static files, we usually don\u0026rsquo;t need to support all HTTP request methods, such as POST, PUT, DELETE, and CONNECT. Only HEAD and GET are enough to serve the files. Nginx has a way to define that you want to deny access to it, unless it is on the list of allowed request methods.\nhttp { server { location / { # Deny access, unless it is GET (HEAD is included with GET) limit_except GET { deny all; } # Other location directives } } } Limit access to sensitive data limit access information leakage Most websites will have a combination of HTML files, CSS and JavaScript. Other file types might be present in a directory structure, especially when using something like WordPress. If we want to restrict access to these files types, we can define a location in the server definition. It should be placed above the other location definitions, so it gets tested first. By using the break keyword, we tell nginx to stop parsing the request if we have a match.\nhttp { server { # Restrict access to some file types location ~ \\.(7z|asp|aspx|bak|bz|bz2|cer|cgi|conf|crt|gz|ini|jsp|key|log|pem|php|php7|rar|sh|sql|tar|txt|zip)$ { return 403; break; } } } Have a good look at the list before deploying it. If you host PHP, then you want to remove that extension most likely from the list. If you host a directory structure with text files (.txt), then remove that as well. The dot itself is escaped, otherwise a request like /do-you-like-martini will be denied as well (.ini).\nBlocking common exploits security The web is a great place, and also for malicious bots that scan your website(s). With some automation they scan the web looking for vulnerable websites. Fortunately it is fairly easy to block many of these common attempts. If you want, you can even go a step further and block repeating offenders.\nThere are a few ways to set up filters to block malicious attempts. We like the method of using a map that compares the requested URI and looks for a match. If there is a match, we then can decide what to do. Let\u0026rsquo;s start with creating a map. If you have just one virtual host, you can define this above the server definition. Another option is to create a separate file that we then include. For this example we use the latter.\nmap $request_uri $is_blocked_common_exploits_path { \u0026#34;~*//\u0026#34; 1; \u0026#34;~*(boot.ini|etc/passwd|self/environ)\u0026#34; 1; \u0026#34;~*(%2e%2e|%252e%252e|%u002e|%c0%2e)\u0026#34; 1; \u0026#34;~*(\\.\\./\\.\\./|\\.\\.\\.|%252e%252e%252e)\u0026#34; 1; \u0026#34;~*(~|`|\u0026lt;|\u0026gt;|:|;|{|}|\\[|\\]|\\(|\\))\u0026#34; 1; default 0; } This map will look at some paths, including double slashes, some system files, double encoded dots, and finally single characters that are often part of file inclusion or path traversal attack. The default 0; sets the value to zero if there is no match.\nIf there is a match found (comparison happens with the $request_uri), then the variable $is_blocked_common_exploits_path will be 1. The next step is to take an action the match.\nhttp { include /path/to/block_common_exploits.conf; server { location { # Blocked URLs from our generic set of common exploits if ($is_blocked_common_exploits_path) { return 403 \u0026#34;Request blocked.\u0026#34;; break; } } } } Note: if you want to apply the rules to all virtual hosts, consider adding it at the highest level.\nConfiguration tests External SSL Labs The well-known SSL server test from SSL Labs can help with testing your SSL configuration. It usually takes a few minutes to complete.\nSecurity Headers The Security Headers website provides a quick way to scan your website and test available response headers.\nAdditional hardening for nginx There is more to do and changes will be made to this guide. Found something that should be included as well? Let it know!\nDefine CSP Set headers Block clients without Accept-Encoding header Limit access logging Create an AppArmor profile Harden nginx systemd service unit Apply the nginx hardening profile for systemd ","permalink":"https://linux-audit.com/web/nginx-security-configuration-hardening-guide/","tags":["certificates","cryptography","hardening","howto","nginx"],"title":"Nginx security hardening guide"},{"categories":["Shell scripting"],"contents":"Want to delete one or more characters from a variable or piped output? There are multiple ways to achieve this using standard system utilities.\nIn this article we use single and double quotes as an example to strip from a variable named myvar. This variable could be filled with something like test'str\u0026quot;ng. With the quotes being special characters, we have to escape them. This way the shell interpreter knows that we mean an actual quote, instead of a string of characters. If you want to test the examples, you could also replace the variable name and put in actual text.\nmyvar=\u0026#34;test\u0026#39;str\\\u0026#34;ng\u0026#34; Remove one or more characters In the examples below we will use multiple tools to replace or delete characters from a variable or piped output from another tool. If you are not sure which tool to select, then have a look at the tr command first. It has a simple delete function to strip out characters. The other next good option is sed as it is powerful and another very common tool.\nUsing AWK With AWK we can use the gsub function to replace multiple occurrences of the string. If we only want to replace the first match, then use sub.\necho \u0026#34;${myvar}\u0026#34; | awk \u0026#34;{ gsub(/[\\\u0026#34;\\\u0026#39;]/, \\\u0026#34;\\\u0026#34;); print }\u0026#34; Another option is reading the first argument of a string that you provide.\nawk \u0026#34;BEGIN{gsub(/[\\\u0026#34;\\\u0026#39;]/, \\\u0026#34;\\\u0026#34;, ARGV[1]); print ARGV[1]}\u0026#34; \u0026#34;mytes\\\u0026#34;ts\u0026#39;tring\u0026#34; This example shows how to use AWK using arguments, which can be useful to replace strings without using echo\n\u0026raquo; Mastering the tool: AWK cheat sheet\nAWK cheat sheet Using perl Another option is using Perl. Especially with its well-known format regular expression, it is easy to replace text.\necho \u0026#34;${myvar}\u0026#34; | perl -pe \u0026#34;s/[\\\u0026#34;\\\u0026#39;]//g\u0026#34; The syntax is very similar to the sed example below, so have a look at the explanation.\nUsing sed Sed is powerful to when it comes to string manipulation. Therefore it is a good option to make changes to an existing string of text.\necho \u0026#34;${myvar}\u0026#34; | sed \u0026#34;s/[\\\u0026#34;\\\u0026#39;]//g\u0026#34; s/: search for a regular expression [pattern]: pattern to sure //: replace the matched pattern with nothing (=delete) g: do this globally, so multiple times Using tr echo \u0026#34;${myvar}\u0026#34; | tr -d \u0026#34;\\\u0026#34;\\\u0026#39;\u0026#34; As one might expect, the -d is short for --delete and removes characters.\nGot another tool that should be listed here as well? Let it know!\nHappy scripting!\n","permalink":"https://linux-audit.com/shell-scripting/strip-one-or-more-characters-from-variable-or-output/","tags":["howto","linux","shell script"],"title":"Strip one or more characters from a variable or output"},{"categories":["Cheat sheets","System Administration"],"contents":"This cheat sheet is intended for beginners and regular users of AWK. At the top it includes the basics and common variables and operators. Near the middle and end there are a lot of examples included to showcase how AWK can be used.\nBasic usage Most one-liners or scripts written in AWK consist of an expression and at least one statement. Typically this is something like if the value in the first field is 10, then show field 2 and 3.\nFormat Intended action {statements} Perform the statement or statements defined within the brackets if (expression) {statements} Perform statement if the logical comparison is true if (expression) {statements1} else {statements2} Execute statements1 group if true, statements2 otherwise If the separator of fields is not a space/tab, then it needs to be defined with -F followed by the separator.\nawk -F: '{ if($1==\u0026quot;root\u0026quot;) {print} }' /etc/passwd\nVariables Within AWK a few variables are used that have a special meaning. They can be used to perform comparisons or to print a result.\nVariable name Usage $0 Full line $1, $2, $3 \u0026hellip; $NF First, second, third, and last field NF Number of Fields NR Number of Records OFS Output Field Separator (default: \u0026quot; \u0026ldquo;) FS Field Separator for input (default: \u0026quot; \u0026ldquo;) ORS Output Record Separator (default: \u0026ldquo;\\n\u0026rdquo;) RS Record Separator for input (default: \u0026ldquo;\\n\u0026rdquo;) FILENAME The name of the file Example showing the number of fields for each line of the passwd file:\nawk -F: '{print NF}' /etc/passwd\nOperators Within AWK it is common to use some kind of operator two compare two values. For example, if value1 is greater than value2.\nOperator Meaning \u0026lt; Less than \u0026lt;= Less than, or equal to \u0026gt;= Greater than or equal to \u0026gt; Greater than == Equal to != Not equal to ~ Match or contains (comparing strings) !~ No match (comparing strings) \u0026amp;\u0026amp; Boolean operator (AND) || Boolean operator (OR) Operators are typically used within an if-statement and decide if a statement needs to be executed.\nThere are also mathematical operators\nArithmetic operator Meaning x + y Addition (2+1=3) x - y Subtraction (5-2=3) x * y Multiplication (2*3=6) x % y Remainder (5%2=1) BEGIN and END Sometimes we want to take an action before we even processed the first line. The BEGIN statement makes this possible. On the opposite END is what performs an action after everything has been processed. This one might be good to summarize information or transform the outcome.\nFrequently used snippets Snippet Intended goal Example snippets BEGIN Perform action before any input is processed Parse /etc/passwd file NR\u0026gt;1 Only show the line after x (1 in this case) Parse output of ss command NR==2 Only show the second line END{print NR} Print the number of records (wc -l) NR%2==0 Show only the even lines $1==\u0026ldquo;a\u0026rdquo; \u0026amp;\u0026amp; $2==\u0026ldquo;b\u0026rdquo; Match only if both expression are valid {a[$2]++}END{for(n in a)print n, a[n]} Count items based on value in field 2, then show number of lines with that value Note: we use short notation here for display, NR%2==0 is probably better written as NR % 2 == 0 in your one-liners\nShowing output Usually we want to display the output, which can be done using print or printf. The first function will simply show the output, while the second can do also some formatting. For example, it can show a textual string and format it into a column of a specific size. It can even strip decimals from floating numbers.\nFormatting output The printf function can be used to format a floating number and limit the number of decimals.\necho 8765.4321 | awk '{printf(\u0026quot;%.2f\\n\u0026quot;,$1)}'\nThe data ($1) comes in via echo and using printf, the number of decimals are reduced to only two. The output will be 8765.43\necho \u0026quot;score=8765.4321\u0026quot; | awk -F= '{printf(\u0026quot;%-16s %.1f\\n\u0026quot;,$1,$2)}'\nData comes in via echo. It needs to be split using the field separator option. We reserve 16 characters for the first field (a text string), then format the number and display with just one decimal\nOutput:\nscore 8765.4 Remove data from some columns Sometimes you may want to show full lines, except one or more columns. This can be done by emptying the column value. For example, if we want to clear out the first two columns, we set both $1 and $2 to an empty string.\nawk -F, '{$1=$2=\u0026quot;\u0026quot;; print $0}' myfile.csv\nCounting results By using a counter, we can easily see the unique number of entries from a file.\nawk '{count[$1]++}; END { for (i in count) print i, count[i] }' /var/log/nginx/access.log\n$1 is the IP address in a default nginx access log.\nWant to count the number of occurrences based on a specific pattern only, then we add an if followed by the counter. At the end we use a for loop to display the results.\nawk '{if ($9~\u0026quot;NextCloud-News/1.0\u0026quot;) { a[$3]++ }} END { for (n in a) print n, a[n] }' file.log\nThis one-liner searches for the user agent in field 9. For every match, it will increase the counter based on field 3. When we are done with processing, we loop through the results after the END.\nSearch a specific pattern If first field equals to pattern, then show full line:\nawk '($1 == \u0026quot;pattern\u0026quot;) {print $0}' filename\nIf the line starts with pattern1 or pattern2, then show the second field:\nawk '($1 ~ /^(pattern1\\|pattern2)/) {print $2}' filename\nUsing environment variables Show the username stored in USER. See the export command for other environment variables that may be available.\nawk 'BEGIN { print ENVIRON[\u0026quot;USER\u0026quot;] }'\nAWK examples In this section we collect examples using variables, operators, and expressions as listed above.\nParse /etc/passwd file The passwd file is formatted properly and has not many surprises in its data output. There are a lot of things we can do with it, so time for some examples.\nLet\u0026rsquo;s start with showing its content and add a line number in front of it:\nawk -F: '{printf \u0026quot;%2s %s\\n\u0026quot;, NR, $0}' /etc/passwd\nOr we could pull in the username and show the user ID, but separate it with a \u0026lsquo;=\u0026rsquo;, possibly for further processing:\nawk -F: '{print $1 \u0026quot;=\u0026quot; $3}' /etc/passwd\nIf we want to search for a particular user account, we can do that as well:\nawk -F: '/root/ {print $3}' /etc/passwd\nWant to return some formatted output and include a nice header? Sure, AWK can do that!\nawk -F: \u0026#39;BEGIN { printf \u0026#34;%-20s %s\\n\u0026#34;, \u0026#34;Username\u0026#34;, \u0026#34;Home directory\u0026#34; printf \u0026#34;%-20s %s\\n\u0026#34;, \u0026#34;--------\u0026#34;,\u0026#34;--------------\u0026#34;} { printf \u0026#34;%-20s %s\\n\u0026#34;, $1, $(NF-1) } \u0026#39; /etc/passwd Output:\nUsername Home directory -------- -------------- root /root daemon /usr/sbin bin /bin How does it work? The BEGIN and the next two printf lines show a header. The first field is formatted by reserving 20 characters of space. Finally, the last printf fills two strings (%s). The first string contains the first field ($1) and represents the username. It is also 20 characters wide, so that longer usernames can fit. Then the second string is filled using the field one left from the last one.\nWe can also transform fields before displaying them. For example, if a user has the /usr/sbin/nologin shell, we can alter the text.\nawk -F: 'BEGIN {OFS = FS}{if($7==\u0026quot;/usr/sbin/nologin\u0026quot;) $7=\u0026quot;Thou Shalt Not Pass!\u0026quot;; print}' /etc/passwd\nTo set the delimiter to a colon like it normally has in this file, we define the Output Field Separator (OFS) to the Field Separator (FS).\nParse output of ss command By default, ss shows an output that is easy to read, but not easy to parse.\nExample: we want to know what ports are in a listening state (TCP) and UDP ports that are open. Let\u0026rsquo;s use the ss -lunt output.\nss -lunt | awk \u0026#39;NR\u0026gt;1{i=split($5,a,\u0026#34;:\u0026#34;);print a[i]}\u0026#39; How does it work?\nNR\u0026gt;1: skip the first line i=split($5,a,\u0026rdquo;:\u0026rdquo;): split our input in field 5 (delimiter is colon) print a[i]: print the last field from each split operation Output:\n53 111 111 53 22 111 22 111 Now with numeric sorting and only show each value once (unique).\nss -lunt | awk \u0026#39;NR\u0026gt;1{i=split($5,a,\u0026#34;:\u0026#34;);print a[i]}\u0026#39; | sort -n -u New output:\n22 53 111 ","permalink":"https://linux-audit.com/cheat-sheets/awk/","tags":["awk","cheatsheet","command-line","howto","linux","one-liner","terminal"],"title":"AWK cheat sheet"},{"categories":["Nginx","System Administration","Web"],"contents":"Nginx usually stores one virtual host per configuration file and each one is configured using the server_name entry. One could create a custom shell script to parse these files, but there is a more reliable method.\nUsing the configuration test option Normally you could test the nginx configuration using the -t option. The capitalized -T does the same, but also shows the configuration. This can be great for showing any configured virtual host and the related hostname(s).\nnginx -T -q | grep server_name This will show the related entries in the configuration, but not as clean as we always want.\nnginx -T -q | grep server_name | awk \u0026#39;{if($1==\u0026#34;server_name\u0026#34;){print}}\u0026#39; | tr \u0026#39; \u0026#39; \u0026#39;\\n\u0026#39; | grep -v \u0026#39;^$\u0026#39; | tr -d \u0026#39;;\u0026#39; | grep -v server_name | sort -u So what this does do?\nnginx: show the configuration grep: only filter out the configured server_name lines awk: only show lines where there is actual configuration of the server_name tr: replace spaces with line breaks grep: strip out empty lines tr: delete the semi-colon grep: strip out the server_name keyword sort: sort and make output unique The output then only shows the configured domains:\narchive.linux-audit.com linux-audit.com www.linux-audit.com Perfect!\n","permalink":"https://linux-audit.com/web/nginx-show-all-configured-virtual-hosts/","tags":["awk","grep","howto","nginx","sort"],"title":"How to see all virtual hosts in nginx"},{"categories":["Security Frameworks","System Administration"],"contents":"AppArmor is the profile-based security security framework and available on many Debian-based distributions. In this article we look at the basics of this security framework, its purpose and features.\nHistory AppArmor was developed by Canonical Ltd. and therefore included in their own Ubuntu distribution. included in Ubuntu and other Debian-based distributions\nWhat is AppArmor? AppArmor is a profile-based MAC framework. As the name implies, profiles are used that focus on simplifying security management. This is done by using application-level confinement. Unlike SELinux, which uses a system-wide policy approach, AppArmor employs profiles that define the allowed behaviors of individual applications or processes.\nMain features Profile-Based Confinement: AppArmor profiles specify the allowed behaviors and access permissions for individual applications, reducing the attack surface and limiting potential security breaches. Simplified Configuration: AppArmor offers a relatively straightforward configuration process, making it accessible to administrators with varying levels of expertise. Integration with Package Management: AppArmor profiles can be automatically generated or augmented based on information provided by package maintainers, streamlining the security configuration process. ","permalink":"https://linux-audit.com/security-frameworks/apparmor/","tags":["apparmor","security framework"],"title":"AppArmor"},{"categories":["Security Frameworks","System Administration"],"contents":"SELinux is a well-known security framework on Linux systems like Red Hat Enterprise Linux. Let\u0026rsquo;s have a look at the basics of this framework, such as its purpose and features.\nHistory SELinux was developed by the National Security Agency (NSA) and integrated into many Linux distributions, including Red Hat Enterprise Linux (RHEL) and CentOS.\nPurpose SELinux is a MAC framework that enforces fine-grained access controls. SELinux achieves this by labeling files, processes, and network ports with security contexts. Additionally, it defines policies that dictate permissible interactions between these entities. Other actions will be blocked.\nSELinux aims to provide strong isolation and confinement of processes. The benefit is reducing the attack surface of the system and mitigating the impact of current and future security vulnerabilities.\nMain features Type Enforcement: SELinux uses a type enforcement model to categorize processes and objects based on security contexts, enabling granular control over access permissions. Role-Based Access Control (RBAC): SELinux supports RBAC, allowing administrators to define roles and assign permissions to users based on their roles. Multi-Level Security (MLS): SELinux supports MLS, enabling different sensitivity levels for data and ensuring strict separation between classified information. ","permalink":"https://linux-audit.com/security-frameworks/selinux/","tags":["selinux","security framework"],"title":"SELinux"},{"categories":["File Systems","System Administration"],"contents":"Changing file permissions: chmod The primary command to change file permissions on a Linux system is chmod. It\u0026rsquo;s a basic system administration utility and pre-installed on the system.\nTo make changes to an existing directory or file, it is first good to look up the existing permissions. This can be done using the ls -l command, that lists them with the long format.\n# ls -l /etc/hosts -rw-r--r-- 1 root root 241 Feb 2 19:10 /etc/hosts There are two syntax styles to tell chmod what the new value should be. So let\u0026rsquo;s look at them both.\nUser, Group, Others If have a directory named docs and we want all users on the system to be able to access it, we grant the Read (r) and Execute (x) permissions.\nchmod o+rx docs\nSo in this example we ask the chmod command to change the permission set of Others (o) and add (+) Read (r) and Execute (x) to the docs directory.\nNumeric values An alternative is using the octal values of the individual file permissions. To set the permissions on the directory so that everyone on the system can access it, we might use the 755 value.\nchmod 755 docs\nNow everyone can access the directory similar to the example above.\nCommon values Most directories and files should be at least readable the the user (owner) and the group that this user belongs to. If a directory or file is not containing sensitive information, then generally the other users will be granted read access. If we represent this in a numeric notation, then this would become 755 (rwx,r-xr-x) for directories and 644 (rw-,r--,r--) for files. The extra \u0026lsquo;x\u0026rsquo; is required to allow users to access the directory.\nFor directories and files that are more strict, such as your SSH configuration files, they typically need a value of 640 or even 600. This way only you (user/owner) can access the file.\nRecursive changes If you want to change all files and directories within a directory, then specify the --recursive (-R) option.\nchmod -R o+r docs\nThis grants the other users read access to the docs directory, including the underlying files and directories.\n","permalink":"https://linux-audit.com/filesystems/file-permissions/how-to-change-file-permissions/","tags":["file permissions","file system"],"title":"How to change file permissions"},{"categories":["File Systems","System Administration"],"contents":"If you want to know which hard links are present, the find utility can give you the answer. In this article we have a look at a few ways to discover more information about hard links.\nGood to know: a hard link shares the same inode, where a symbolic link has its own inode and just points from one to another.\nShow all hard links within a specific file system or directory When we have a directory with hard links, we can discover by looking at the link count of each file. If it has more than 1 link, then there must be a hard link present.\nfind . -xdev \\! -type d -links +1 -printf \u0026#39;%40p --\u0026gt; inode %i\\n\u0026#39; | sort If you want to use this information and parse it in an easier way, change the printf into \u0026rsquo;%i=%p\\n\u0026rsquo;.\nIn the example above we only searched in the current directory and below. You can change this into a specific file system. The -xdev prevents searching on external file systems (e.g. NFS).\nShow other files that link to the same inode or file There are a few ways to use find to also look the related files that link to a specific inode. This is the specific unique pointer stored in a file system. To show the inode, use the \\-i option:\nls -li /etc\nThe first column listed is the inode.\nTo see which files are pointing to this inode, we can use the find command and define what inode we are looking for.\nfind . -inum 123456\nAnother option is to specify the file name itself. In other words, we ask find to lookup the inode and do the same step as above, but simplified.\nfind . -samefile /path/to/the/file\nIf you suspect that there are hard links to the same file outside the current work directory, then provide the full file system.\n","permalink":"https://linux-audit.com/filesystems/show-hardlinks-or-multiple-links-to-the-same-file/","tags":["file permissions","file system","inode","ls"],"title":"How to find hard links or files that point to a specific file"},{"categories":["File Systems","System Administration"],"contents":"Every file that is stored has a set of file permissions stored within the filesystem. This data about the actual data, is called meta-data. Let have a look at how file permissions work on Linux systems and how to read and understand them.\nRead, Write, and Execute Linux file permissions are divided into three main categories:\nRead (r): Allows users to view the contents of a file or directory Write (w): Grants users the ability to modify the contents of a file or directory Execute (x): Enables users to execute a file or access the contents of a directory User, Group, and Others These permissions are each assigned to three entities:\nUser (u): The user who owns the file or directory, or the owner of the file Group (g): A collection of users who share common permissions Others (o): Everyone else who is not the owner or a member of the group So in total we have three sets of file permissions, one for these three entities.\nRepresentation of the file permissions In Linux, permissions are represented by a series of ten characters. The first character indicates the type of file (e.g., regular file, directory, or symbolic link). The remaining nine characters represent the permissions for the owner, group, and others. These nine characters are grouped into sets of three, each indicating the read, write, and execute permissions, respectively.\nFor example, the permission string drwxr-xr-- can be explained as:\nd: Directory rwx: Read, write, and execute permissions for the owner. r-x: Read and execute permissions for the group. r--: Read-only permissions for others. Numeric representation While the symbolic representation of permissions is intuitive, Linux also offers a numeric representation that simplifies permission management. Each permission is assigned a numeric value:\nPermission Abbreviation Octal value Read r 4 Write w 2 Execute x 1 So how do we use these values? We simply add up the values of the individual permissions.\nrwx (read, write, execute) = 4 (read) + 2 (write) + 1 (execute) = 7 rw- (read, write) = 4 (read) + 2 (write) = 6 r-x (read, execute) = 4 (read) + 1 (execute) = 5 r-- (read-only) = 4 (read) Let\u0026rsquo;s have a look at a practical example. Suppose we have a file named example.txt with the following permissions: -rw-r--r--\nThe numeric representation would be 644, as the owner has Read(4) + Write (2), the Group and Others only Read (4).\nIf you don\u0026rsquo;t want other users to have access to this file, you can use chmod to change the file permissions.\nchmod 640 example.txt\nHere are all the value combinations:\nPermissions Octal Value --- 0 --x 1 -w- 2 -wx 3 r-- 4 r-x 5 rw- 6 rwx 7 With this introduction into file systems, it is time to move to the next step!\n","permalink":"https://linux-audit.com/filesystems/file-permissions/introduction-to-linux-file-permissions/","tags":["file permissions","file system","linux"],"title":"Introduction in Linux file permissions"},{"categories":["Software","Web"],"contents":"Fresh look at RSS after a migration This blog had a RSS feed since its inception about 10 years ago. It was (and is) an easy way for readers to quickly discover released and updated articles. Although a lot has changed in 10 years, including a migration from WordPress to Hugo, the RSS feed is still available. Recently, as part of the migration, we looked again at all individual layers that makes this blog possible. From the web server configuration, up to the final HTML output, everything got a review.\nInstead of just copying the old configuration, we set everything up from scratch. A fresh start, questioning all choices. With each change, we looked what we could tune and improve. Things that could improve availability and performance. For example, the SSL/TLS configuration settings were updated, including enabling 0-RTT handshakes. The blog was already somewhat static and quick, but there was still room for improvement. This time everything is really static output and we let the web server focus on what it is good at: delivering content at a high speed! Upon on our analysis we discovered a few things, and that is what this article is about.\nBad bots Like every website on the internet, our logs getting spammed with bad bots. We already had some measures implemented, but we decided to optimize this even further. So this means not just blocking bad bots, but also blocking badly behaving clients. Using still the HTTP/1.0 protocol? That\u0026rsquo;s fine, but not on this website. Not offering to accept compressed data transfers, sorry, no data for you. After rejecting some of these requests, that is where things got interesting!\nOur rationale In the initial version of this blog post, we did not mention the rationale behind this blog post. So let\u0026rsquo;s have a look at that first before showing some examples.\nReduce the amount of unnecessary traffic Block clients that misbehave (on purpose or by accident) Become more sustainable with our digital resources and assets Remove any clutter from our log files, to easier monitor requests Increase our security posture Help the community Inform about this \u0026ldquo;invisible\u0026rdquo; issue Pointing out incorrectly configured clients Reporting the issues to the (open source) projects That being said, let\u0026rsquo;s have a look together at some of the things we recently observed, including our thoughts. As we are in favor of RSS, we will also add the relevant actions that we took to see if things can be improved. Not just for us, but for the whole RSS community. During our journey, we already encountered some negative responses to reporting the issues. This article and all actions are written with the best intention in mind.\nExamples of issues and improvements Different types of requests from Slackbot 2024-04-13T11:56:26+00:00 200 1.2.3.4 \u0026#34;HEAD /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Slackbot 1.0 (+https://api.slack.com/robots)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-13T11:56:26+00:00 200 2.3.4.5 \u0026#34;GET /feed/ HTTP/1.1\u0026#34; 19046 \u0026#34;-\u0026#34; \u0026#34;Slackbot 1.0 (+https://api.slack.com/robots)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . This is interesting. It looks like Slack first queries some basic details about the feed by using a HEAD. The assumption was that this is feedback for decided to pull in the feed (or not). However if we look at the timing, we see something else. In the very same second that the first request came in, another system does an actual GET. I doubt they got the chance to process the information from the first request before firing up the second. Bad implementation? Not sure. Another interesting thing is that the used TLS protocol and ciphers are the same, but the HEAD request was done with an older HTTP protocol version. Might be a thing related to reducing overhead?\nMultiple requests from the same system Some clients seem to request the feed a few times per minute.\n2024-04-13T11:58:13+00:00 200 1.2.3.4 \u0026#34;GET /atom.xml HTTP/1.1\u0026#34; 14697 \u0026#34;https://linux-audit.com/\u0026#34; \u0026#34;Inoreader/1.0 (+http://www.inoreader.com/feed-fetcher; 1 subscribers; )\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-13T11:58:29+00:00 200 1.2.3.4 \u0026#34;GET /atom.xml HTTP/1.1\u0026#34; 14697 \u0026#34;https://linux-audit.com/\u0026#34; \u0026#34;Inoreader/1.0 (+http://www.inoreader.com/feed-fetcher; 1 subscribers; )\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . While this client only had two requests, it is a waste of 50 percent. After all, nothing changed in this short time. It\u0026rsquo;s not fully clear why this client did this, especially as it is not continuously doing this. If it had, our rate-limit would kick in.\nStatus: cause unclear, more research needed Actions:\nNone, more research needed to see if this is a one-time event or common issue Newsboat: Too many requests With rate-limiting in place, we noticed that the Newsboat client got picked up. Example from the logs:\n2024-04-14T09:07:39+00:00 304 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Newsboat/r2.35 (Linux x86_64)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-14T09:07:39+00:00 304 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Newsboat/r2.35 (Linux x86_64)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-14T09:07:39+00:00 304 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Newsboat/r2.35 (Linux x86_64)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-14T09:07:39+00:00 304 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Newsboat/r2.35 (Linux x86_64)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-14T09:07:40+00:00 429 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 74 \u0026#34;-\u0026#34; \u0026#34;Newsboat/r2.35 (Linux x86_64)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . These requests were interesting, as a 304 was reported for the first four, then followed by a 429 error. The HTTP 304 status means that content was most likely not modified compared with the copy that the client has. This is done by comparing the last-modified header. So bonus points for implementing this, as this saves a lot of unneeded data traffic. With zero bytes being sent, that is a perfect outcome. At the same time, we see that multiple requests are made in the same second. So in the end, the client and web server are still processing useless requests. When the rate-limit kicks in, a 429 status is returned and the conversion stops. That is, until the next set of requests.\nStatus: waiting for new Newsboat release Actions:\nCreated issue Issue acknowledged Should be resolved with pull request The issue was reported and one of the developers quickly picked it up. Awesome! This one needs to be monitored and hopefully next release will no longer be responsible for unneeded requests.\nSelfoss: Not supporting data compression and multiple retries The next one is Selfoss. We have seen it showing up in the logs and nothing special so far. Until we toggled the switch to disallow requests that don\u0026rsquo;t support compression (accept-encoding header).\n2024-04-13T10:05:26+00:00 426 1.2.3.4 \u0026#34;GET /atom.xml HTTP/1.1\u0026#34; 16 \u0026#34;https://linux-audit.com/atom.xml\u0026#34; \u0026#34;Selfoss/2.19 (+https://selfoss.aditu.de)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-13T12:02:24+00:00 426 2.3.4.5 \u0026#34;GET /feed/ HTTP/1.1\u0026#34; 16 \u0026#34;https://linux-audit.com/feed/\u0026#34; \u0026#34;Selfoss/2.19 (+https://selfoss.aditu.de)\u0026#34; TLSv1.2/ECDHE-ECDSA-AES256-GCM-SHA384 0.000 . So here we have two different clients, about two hours apart from each other. One uses the /atom.xml link, the other the alias /feed/. Same file, different path. Both clients use a different set of TLS protocol and ciphers, so I assume that has to do with the underlying operating system and libraries. The Selfoss software itself looks to be the same when looking at the version number. The used HTTP protocol is also the same. Maybe the HTTP/1.1 is somewhat outdated, but that\u0026rsquo;s fine.\nThe interesting part in this case is that the requests both got blocked. This can be seen as we returned a 426 message, telling the client to upgrade. As it is not due to the HTTP protocol version, it is related to the lack of compression support.\nAnother issue that was found\nStatus: issue most likely solved (depends on Guzzle) Actions:\nOpen an issue on GitHub Project implemented changes to improve this, including upstream to the Guzzle (PHP HTTP client) Created new issue (2024-12-27) for repeating errors instead of handling it on the client side With the actions taken by the project, most likely this issue will be resolved in the upcoming update. That\u0026rsquo;s awesome!\nFeedbin: Sometimes supporting date compression? Like the example with Selfoss, we came also across clients that behave differently per request.\n2024-04-13T12:06:35+00:00 200 1.2.3.4 \u0026#34;GET /feed/ HTTP/1.1\u0026#34; 17863 \u0026#34;-\u0026#34; \u0026#34;Feedbin feed-id:MASKED - MASKED subscribers\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-13T12:06:36+00:00 426 1.2.3.4 \u0026#34;GET /web/nginx-log-only-some-requests/ HTTP/1.1\u0026#34; 16 \u0026#34;-\u0026#34; \u0026#34;Down/5.4.1\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . In this example we see that the RSS feed is pulled in. A second later, the latest blog post is retrieved. It came from the same IP address, but with a different user agent. The HTTP protocol, TLS protocols, and ciphers, all the same. So probably different components are at work. One that tracks RSS feeds, while the other pulls in the data related to the article? Not sure what it is and this needs more research.\nStatus: more research needed Actions:\nNone so far, need more samples Tiny Tiny RSS: Not all versions supporting data compression 2024-04-20T18:25:59+00:00 304 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Tiny Tiny RSS/21.07-73d14338a (http://tt-rss.org/)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-20T22:48:30+00:00 304 2.3.4.5 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Tiny Tiny RSS/22.09-d47b8c8 (https://tt-rss.org/)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-21T04:53:18+00:00 304 3.4.5.6 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 0 \u0026#34;-\u0026#34; \u0026#34;Tiny Tiny RSS/23.09-f489f620 (https://tt-rss.org/)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-21T04:56:30+00:00 426 4.5.6.7 \u0026#34;GET /feed/ HTTP/1.1\u0026#34; 16 \u0026#34;-\u0026#34; \u0026#34;Tiny Tiny RSS/24.03-435c321ca (https://tt-rss.org/)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . After looking in the code, it seems that Tiny Tiny RSS is using Guzzle as its HTTP client. But wait, that is the same component as in Selfoss! So it might be possible that without any changes to Tiny Tiny RSS, it will inherit the changes.\nUpdate: later on, we noticed that earlier versions of TT RSS did actually use the Accept-Encoding header (and even use the modern HTTP/2.0 protocol instead of HTTP/1.1).\nStatus: issue most likely solved (no further action) Actions:\nCreated bug report at their community forum Received responses from two developers Developer wn_name confirmed they switched to Guzzle , which explains why the older version performs a different request Updated log entries to show some older and newer versions I hope the project also considers to check out if Guzzle can do HTTP/2.0 requests in the future to further optimize the performance. As one of the replies (about blocking clients that not offer data compression) was \u0026ldquo;tells me he’s some kind of self-important internet weirdo which i’d rather not do anything for.\u0026rdquo;, I believe any other feedback is not very welcome at the moment. Cased closed.\nMiniflux: supporting Gzip, but not Brotli (resolved) In the log we also discovered different file sizes for the feed. Example of a few requests:\n2024-04-13T13:30:02+00:00 200 1.2.3.4 \u0026#34;GET /feed/ HTTP/1.1\u0026#34; 17183 \u0026#34;-\u0026#34; \u0026#34;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-13T13:31:22+00:00 200 2.3.4.5 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 20621 \u0026#34;-\u0026#34; \u0026#34;Mozilla/5.0 (compatible; Miniflux/2.1.2; +https://miniflux.app)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . The first request looks to be a normal browser, while the second one is another RSS reader named Miniflux. It already uses compression, but it needed more data traffic to receive the same file. When we look at the disk, we can see the related values for the feed (at that very moment, it changes daily).\n-rw-r--r-- 1 www-data www-data 72474 Apr 13 13:18 atom.xml -rw-r--r-- 1 www-data www-data 17183 Apr 13 13:18 atom.xml.br -rw-r--r-- 1 www-data www-data 20621 Apr 13 13:18 atom.xml.gz This RSS reader is already a good job. It uses a modern HTTP protocol version and has data encoding implemented. BY using a different compression method, it could save (in this case) 3438 bytes. That doesn\u0026rsquo;t sound like a lot, but we limited the number of entries in our feed. There are many more feeds that are much bigger in size and then the differences add up.\nStatus: improved (waiting for new release) Actions:\nOpened a feature request Change has been made with an existing pull request and new PR to add Brotli! 2024-04-19: Brotli support added, now waiting for next release to have this active\nNextcloud News App Also the Nextcloud News app has interesting behavior: it opens an initial connection to the feed, then pulls in a number of URLs. Seeing that these are recently changed pages, it becomes clear that these are coming from the feed.\nWe initially saw this with user agent \u0026lsquo;NextCloud-News/1.0\u0026rsquo;, but still see it also with newer ones like \u0026lsquo;NextCloud-News/25.1.0\u0026rsquo;. So looks like it is still missing data encoding support on the additional HTTP requests after the initial feed retrieval.\n2024-12-14T14:30:24+00:00 200 1.2.3.4 \u0026#34;GET /feed/ HTTP/1.1\u0026#34; 417 \u0026#34;-\u0026#34; \u0026#34;NextCloud-News/25.1.0\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-12-14T14:30:24+00:00 406 1.2.3.4 \u0026#34;GET /system-administration/commands/pidstat/ HTTP/2.0\u0026#34; 55 \u0026#34;-\u0026#34; \u0026#34;NextCloud-News/1.0\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-12-14T14:30:24+00:00 406 1.2.3.4 \u0026#34;GET /system-administration/commands/apt-file/ HTTP/2.0\u0026#34; 55 \u0026#34;-\u0026#34; \u0026#34;NextCloud-News/1.0\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-12-14T14:30:24+00:00 406 1.2.3.4 \u0026#34;GET /system-administration/commands/vmstat/ HTTP/2.0\u0026#34; 55 \u0026#34;-\u0026#34; \u0026#34;NextCloud-News/1.0\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-12-14T14:30:24+00:00 406 1.2.3.4 \u0026#34;GET /website/latest-changes/ HTTP/2.0\u0026#34; 55 \u0026#34;-\u0026#34; \u0026#34;NextCloud-News/1.0\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-12-14T14:30:24+00:00 406 1.2.3.4 \u0026#34;GET /system-administration/commands/systemd-analyze/ HTTP/2.0\u0026#34; 55 \u0026#34;-\u0026#34; \u0026#34;NextCloud-News/1.0\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . The strange behavior is that the feed is pulled in with another HTTP protocol version. Also, the initial request uses data compression, yet the other 10 seconds are not. We are refusing to waste bandwidth if not needed, so they are blocked with a 426 message.\nStatus: most likely improved (monitoring) Actions:\nOpened a feature request Similar issue was reported, which looks to indicate that older software is having this issue Feed on Feeds (with SimplePie dependency) Another tool using a different set of protocols, with the initial request allowing data compression, yet the other two not.\n2024-04-18T18:12:43+00:00 200 1.2.3.4 \u0026#34;GET /feed/ HTTP/2.0\u0026#34; 21552 \u0026#34;https://linux-audit.com/feed/\u0026#34; \u0026#34;FoF SimplePie/1.5.6 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20230917075900\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-18T18:12:43+00:00 426 1.2.3.4 \u0026#34;GET / HTTP/1.1\u0026#34; 16 \u0026#34;-\u0026#34; \u0026#34;FavIcon/1.0 (Caching Utility; ; Allow like Gecko) Build/20160424000000\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-18T18:12:43+00:00 426 1.2.3.4 \u0026#34;GET /favicon.ico HTTP/1.1\u0026#34; 16 \u0026#34;-\u0026#34; \u0026#34;FavIcon/1.0 (Caching Utility; ; Allow like Gecko) Build/20160424000000\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . Sometimes it looks like projects use different clients to pull in data. But it may also be as simple that the request to an underlying library is getting different parameters. Let\u0026rsquo;s see!\nStatus: waiting for response on reported issue Actions:\nOpened an issue . Closed. Seems that it is not SimplePie, but another project (Feed-on-Feeds) that uses an older version of SimplePie, so created a new issue No response to the reported issue (Dec 2024), but also no longer seeing the agent in the log files Feedly (fixed) With some AWK magic, I found also a consumer of the feed that apparently does not such headers to see if the feed was changed at all.\n2024-04-22T12:11:14+00:00 200 1.2.3.4 \u0026#34;GET /atom.xml HTTP/2.0\u0026#34; 51797 \u0026#34;-\u0026#34; \u0026#34;FeedlyBot/1.0 (http://feedly.com)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-22T12:11:21+00:00 200 1.2.3.4 \u0026#34;GET /atom.xml HTTP/2.0\u0026#34; 51797 \u0026#34;-\u0026#34; \u0026#34;FeedlyBot/1.0 (http://feedly.com)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . 2024-04-22T12:11:32+00:00 200 1.2.3.4 \u0026#34;GET /atom.xml HTTP/2.0\u0026#34; 51797 \u0026#34;-\u0026#34; \u0026#34;FeedlyBot/1.0 (http://feedly.com)\u0026#34; TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 . Actions:\nContacted them by email (2024-04-22) Resolved (2024-05-15) Solution: They changed the feed URL on their side. When a redirect (301) happens, the state is not correctly stored, resulting in a 200 after the redirect. With the feed URL change on their end, things should be fine now.\nConclusion While most RSS feed readers seem to continue to work properly after adjusting our web server configuration, a few unexpected issues came up. So far, multiple open source projects used these insights to improve and made changes right away. Where one project started to support Brotli compression, others enabled the required Accept-Encoding header to enable data compression. Awesome!\nThe general attitude towards the suggestions made was very positive, from readers of this blog post, up to several developers. So far there is only one negative experience, but maybe it was to be expected that how much you try to do something good, there is always someone having a bad day.\nWhat\u0026rsquo;s next? In the upcoming months we will continue to monitor our log and specifically look at the RSS feed. Hopefully more clients can be upgraded to use modern protocols, content encoding, and reduce the number of requests by using the Last-Modified header. When we receive updates, this blog post will be updated.\nI also received a private message with an interesting RSS issue tracker . Might be worth checking now and then!\nTips for a better RSS community Developers of RSS readers Implement the usage of the headers If-Modified-Since or If-Unmodified-Since to leverage the Last-Modified status Use data encoding methods to reduce the file size that needs to be sent Implement HTTP/2.0 protocol where possible Publishers of RSS feeds Reduce the number of entries in the feed (based on your publish frequency). More is not always better. Compress your feeds, when possible pre-compressed and with different methods (e.g. Gzip and Brotli) Implement HTTP/2.0 protocol where possible Consider implementing rate-limiting to indicate misbehaving clients Make your feed cacheable When possible, provide a Last-Modified header with a value that is consistent When possible, set the Expires header (or Cache-Control) When possible, provide an ETag header for use with If-None-Match Tips for users of RSS feeds If you want to be a good net citizen, reduce the time that you refresh your feeds. Is it really needed to request them multiple times a day? Consider refreshing the ones that are not updated daily with a lower interval than those that are updated daily. Also check if your RSS reader is up-to-date, especially after several improvements have been implemented.\nFeedback? Got something to share based on the results that we saw recently? Let it know!\n","permalink":"https://linux-audit.com/rss-is-cool-but-some-rss-feed-readers-are-not-yet/","tags":["awk","blog","brotli","gzip"],"title":"RSS is cool! Some RSS feed readers are not (yet)..."},{"categories":["Nginx","Web"],"contents":"Nginx is flexible when it comes to what should be logged in the access.log. With the combination of a map and if-statement, this can be achieved very easily!\u0026quot;\nLog only some events by HTTP status Creating a map using $status The $status variable contains the HTTP status code that is normally returned to each request. We can leverage this status code to set a so-called boolean (true/false, or 1/0). Let\u0026rsquo;s define first the map and use the HTTP status.\nmap $status $loggable { ~^[23] 0; 444 0; default 1; } So this example has a few options in it. Every status code that is considered good is typically in 200 or 300 range (200=OK, 301=Redirect, etc). By using a regular expression we can state that if the status code starts with a 2 or a 3, it is good. Let\u0026rsquo;s say that we are not interested in those for our custom logging. We give it therefore a value of zero.\nIf we want to define another specific value outside this 2xx/3xx range, like 444 (no response), we can define that as well. Again, we don\u0026rsquo;t want to log those, so we give it also a value of zero. All other responses (401, 403, 404, 5xx, etc) are not defined, so they will be caught by our catch-all, which is part of the keyword default. In this case it will get a value of one, as we do want to log them.\nNext step is defining our access log.\nDefine access_log with if-condition Like in a normal configuration, we have to use the access_log statement to define where to log our requests. But in this case we add an if-statement to it. The if=$loggable will only \u0026ldquo;activate\u0026rdquo; the line when it receives a positive value.\nserver { access_log /var/log/nginx/custom.log combined if=$loggable; access_log /var/log/nginx/access.log; } In this piece of configuration we defined our custom log, where we only store those requests with the status codes that we are interested in. In the line below, we define our access_log again, yet this time without any conditions. This way we can still store all events there, for something like log file processing or for other purposes. Our custom.log will be much smaller, as most of the events won\u0026rsquo;t be stored in there.\n","permalink":"https://linux-audit.com/web/nginx-log-only-some-requests/","tags":["howto","logging","log files","nginx"],"title":"How to log only some requests to a log file in nginx"},{"categories":["Shell scripting"],"contents":"When you create a shell script, many things can go wrong. With a few basics you can catch errors easier and at the same time make your scripts (more) fail-safe. The beauty of shell scripting is that with just a few steps this can be achieved!\nEmpty variables: nounset (-u) A very typical issue in shell scripts is an incorrect or empty variable. Usually this happens due to a typo, but sometimes also assignments can be wrong. We can let our shell script stop immediately if this happens. This is a great safety measure during development, but also when a shell script is tested and working its duties in production.\nLet\u0026rsquo;s have a look what happens when you use an empty variable normally:\n#!/bin/sh echo \u0026#34;${oops}\u0026#34; The output is an empty line. Very innocent when it is just some text (not) being displayed. But this can change quickly when the variable is input for a task that uses rm.\nTo enable this measure, put the following line at the start of your script.\nset -o nounset\nSo usually your script would then look like this:\n#!/bin/sh # safeguards set -o nounset echo \u0026#34;${oops}\u0026#34; When we now run the script, it will give a different output.\n./empty-variable.sh: 6: oops: parameter not set So on line 6 we have a parameter not set, which is named oops. In other words, we used a parameter that had not a value assignment.\nWe can also define this nounset option by using the shorter set -u. While this is often used in scripts, it may not always be clear what it does. The fully written options are somewhat more self-documenting, especially for others reading your scripts.\nExit upon errors in last command in pipe: pipefail During the execution of your shell script, a combination of commands will be used. Sometimes single commands, sometimes piped after each other. To quit immediately if something goes wrong in the last part of the pipe, we can use the pipefail option. Let\u0026rsquo;s have a look at a simple script.\n#!/usr/bin/env bash set -o nounset set -o pipefail echo \u0026#34;test 1\u0026#34; | wc -l echo \u0026#34;test 2\u0026#34; | wcc -l echo \u0026#34;test 3\u0026#34; | grep test The pipefail option is usually not available when using /bin/sh as shell\nSo like before we use the nounset to counter for empty variables. Additionally we also set the pipefail to monitor for the exit state. But what if we didn\u0026rsquo;t use it? The output would look like this:\n1 ./error-in-pipe.sh: 7: wcc: not found test 3 Only the first and third echo give the expected result. Again, an innocent echo of text won\u0026rsquo;t make a huge difference. But one wrong replacement or failed action, and data or files could be corrupted, moved, or worse.\nIf we have the pipefail option enabled, then suddenly we see just two lines of output:\n1 ./error-in-pipe.sh: 7: wcc: not found So the first echo goes well, but the second does not. The third won\u0026rsquo;t be performed, as we stop execution when the first (unexpected) issue showed up.\nExit upon errors: errexit (-e) The next step is catch exit states that are not zero. For example when you try to do something with a file that does not exist.\n#!/bin/sh cat /tmp/randomfilethatdoesnotexist echo \u0026#34;Happy end!\u0026#34; When running the script, it will show the following output:\ncat: /tmp/randomfilethatdoesnotexist: No such file or directory Happy end! As you can see, this will return an error followed by \u0026lsquo;Happy end!\u0026rsquo;. Typically, we don\u0026rsquo;t errors and it may have been better to stop upon a serious error to prevent issues.\nSo let\u0026rsquo;s add some safeguards and insert the nounset and errexit the option to the script:\n#!/bin/sh set -o errexit set -o nounset cat /tmp/randomfilethatdoesnotexist echo \u0026#34;Happy end!\u0026#34; The script will now only show the failure and exit immediately after it:\ncat: /tmp/randomfilethatdoesnotexist: No such file or directory Note: the nounset obviously has no influence in this example. It\u0026rsquo;s just here to show that you can combine them.\nWant to learn more about the options that set can provide? Have a look at the GNU Set Builtin page .\nHappy scripting!\n","permalink":"https://linux-audit.com/shell-scripting/making-scripts-more-secure-and-safe/","tags":["linux","shell script"],"title":"Making scripts (more) secure and safe"},{"categories":["Web"],"contents":"Gzip Gzip is well-known and around for some time. Almost all web clients support it and the savings are typically very high.\nCompressing a file with gzip is simple, especially with the command often already installed on Linux systems.\ngzip --best --force filename.html\nThis compresses the file as good as it can (\u0026ndash;best) and overwrite a .gz file if it already exists (\u0026ndash;force).\nBrotli Brotli is a fairly new kid on the block when it comes to compressing files. So besides having Gzip, it is a good idea to pre-compress files also with Brotli. The files are generally smaller than those compressed with Gzip.\nNote: Typically the brotli tool is not installed by default. Use your package manager to see if it is available. Otherwise download it via the Brotli project on GitHub.\nAfter installing Brotli, it is time to run it against a file:\nbrotli --best --force filename.html\nThis command will compress our HTML file and apply the best possible compression using \u0026ndash;best. The \u0026ndash;force will overwrite the compressed file if it already exists. This might be useful when the original file has been changed and needs to be compressed again.\nIf you want to automate things and pre-compress a whole directory of static files, consider using find. Then search for files and apply the compression to those files.\nExample of a find command to apply the changes to recently changed files.\nfind /home/web/linux-audit.com/public/ -type f -mmin -15 -regextype posix-extended -iregex \u0026#39;.*\\.(css|html|js|json|txt|xml)\u0026#39; -exec brotli --best --force {} \\+ That is a line full with things, right? Let\u0026rsquo;s have a look at the individual parts of this command.\nFind option Action -type f Only search for files -mmin -15 Only files that are changed in the last 15 minutes -regextype posix-extended Select specific type of regular expression format \u0026lsquo;.*.(css html -exec brotli \u0026ndash;best \u0026ndash;force {} + Execute a task on the files that have been found Using it in a nginx configuration If you want to use pre-compressed files, it is important to configure nginx correctly. That starts with making sure you got the right support. For example, a modern Ubuntu system might have a pre-compiled module available that can be installed.\napt install libnginx-mod-http-brotli-static\nNext step is adding the support for static files to your /etc/nginx/nginx.conf in the http directive the following:\n# Added brotli/gzip support for static pre-compressed files brotli_static on; gzip_static on; Test if compression is available Using the curl tool we can easily perform a request and tell the web server that we like to receive back a compressed version of the page.\ncurl --head --header \u0026#39;Accept-Encoding: br\u0026#39; https://linux-audit.com/ Check How to test if a website supports Brotli or Gzip for additional tips\nGot other suggestions for pre-compressing files? Let it know!\n","permalink":"https://linux-audit.com/web/how-to-pre-compress-static-assets/","tags":["brotli","gzip","nginx","web server"],"title":"Pre-compress static assets with Brotli and Gzip"},{"categories":["System Administration"],"contents":"Sometimes a process gets stuck and how often you try, it won\u0026rsquo;t respond to the combination of CTRL+C. One option is to open a second shell, then use the kill command followed by the process ID (PID).\nkill 1234\nPushing a job to the background While this works, there is usually a much easier way. This involves pushing a running process into the background by pressing CTRL+Z.\n[1]+ Stopped ./runserver Kill the process To get it back to the foreground, we would normally run fg. Instead, we tell it to stop.\nkill %1\nThe %1 refers to the first process that is in the background, noted by the [1] in the output above. After running the kill command, we see the following output:\n[1]+ Stopped ./runserver It is typically showing the same output, but this time the process received a kill signal. When you put multiple processes to the background, it would make sense to first check the output of the jobs command. It will show any job that is running, including those that are pushed to the background and no longer visible on screen.\nJobs option Explanation -l Show process ID (PID) -n Only show those processes that have a changed status since last run of jobs -p Show PID only -r Only show running jobs -s Only show stopped jobs Do you have another great tip to deal with processes? Let it know!\n","permalink":"https://linux-audit.com/kill-a-process-that-does-not-respond-to-ctrl-c/","tags":["kill","processes"],"title":"Kill a process that won't respond to CTRL+C"},{"categories":["Shell scripting"],"contents":"Shell scripts can be powerful for automation. Sometimes, we want to ask the user for input. Let\u0026rsquo;s have a look at a few options that can be very handy for your next shell script!\nFill a variable with input provided by the user If we like the user to provide us with some details, like a name, email address, or hostname, we can use the read command.\nread -p \u0026#34;What is your name? \u0026#34; name echo \u0026#34;Your name is: ${name}\u0026#34; How does it work? The read -p asks for user input and stores the result in the variable $name. On the second line we use this to display the value provided.\nPrompt for a Yes/No answer while true; do read -p \u0026#34;Do you want to continue? \u0026#34; yesno case $yesno in [Yy]*) echo \u0026#34;You entered Yes!\u0026#34; # Insert here a task to do break ;; [Nn]*) # Exit the program, as user answered \u0026#34;No\u0026#34; exit ;; *) echo \u0026#34;Please answer Yes (y) or No (n).\u0026#34; ;; esac done So how does this work?\nStep 1: Wait The while true makes the program go into a loop.\nStep 2: Ask for user input and process it As there is a read -p directly after the loop, it will wait for user input and put the result in the variable $yesno. If that is a Y (or y), it will perform the step related to that. The asterisk behind the [Yy] allows the user to input Yes, or Yup, or Yeah, as long as it starts with a small or capital Y.\nStep 3: Perform the action After the choice has been made, the action will be performed. If we answer N/n/No/no/Nope/NOK/etc, the program will stop as there is an exit command. In case the answer was something with a Y/y, then we first will perform the action, followed by a break. This break stops the while true loop, otherwise we get stuck into that.\n","permalink":"https://linux-audit.com/shell-scripting/prompt-for-user-input/","tags":["linux","shell script"],"title":"Prompt for user input in a shell script"},{"categories":null,"contents":"We want our blog to provide much value to our readers. With our origins in open source, we share almost everything that we know. At the same time we want it to be a good experience, so easy to read, quick to load, and not invading any privacy. For that reasons, we made some extreme decisions a while back. No tracking and no cookies are just a few examples. With a changing climate we want to go a few steps further. So while overhauling this blog in 2024, we also focused on being more sustainable and applying eco-friendly measures wherever we can.\nFocus on reducing CPU cycles Every action costs an amount of energy, from development up to you as reader looking at this page. Let\u0026rsquo;s have a look at some of the following measures that we implemented:\nStatic website generator Reduce number of systems Switching from WordPress to Hugo In the past we used WordPress to manage and present our blog. Now we use Hugo , a static website generator. Instead of running a combination of PHP and a MariaDB instance 24/7, we now only run the generator when changes were made to the blog. This gives a huge reduction in memory usage and CPU cycles required to keep powering the website.\nDevelopment systems Our development systems are also optimized to limit their CPU usage, including running tasks on low-energy devices. Where possible we consolidated systems, unless this was in conflict with security principles that we apply. Some higher risk environments will be separated, but otherwise consolidation is one of the ongoing objectives.\nReducing bandwidth System fonts Compressing assets Increase caching Rate-limit bots Block outdated or misbehaving systems WebP images Using default system fonts This blog now uses system fonts only. By using a so-called font stack that only contains fonts that are commonly available, readers will get typically a font that is optimized for reading text. The additional benefit is speed, as no external font has to be downloaded and processed.\nCompressing assets Most file formats can be compressed. The result is much smaller data transfers, at the cost of some CPU cycles to compress and decompress. In the past we let our web server compress almost all files (except images) using Gzip compression. That means that the web server was doing this so called \u0026lsquo;on-the-fly\u0026rsquo; compression all day. Even for files that were requested thousands of times a day. Nowadays we pre-compress all files and then offer it when a client supports it. For maximum coverage we now support both Brotli and Gzip. This one-time compress action will be done using the best possible compression options, to save bandwidth as much as possible.\nIncrease caching possibilities In the past we had somewhat limited possibilities when it came to caching. So with the 2024 overhaul, we simplified the configuration of our web server yet enhanced caching settings. Based on the content type (MIME types), we provided a hint to clients how long they should cache resources. This way less requests have to be made over the long term.\nRate-limits With the overhaul we implemented more strict rules when it comes to our bandwidth. Our traffic pool is big, but that doesn\u0026rsquo;t mean we can be more progressive in reducing it further. Clients (including bots) that misbehave, get rate-limited. The response returned is HTTP 429 with a message indicating that rate-limiting was applied. When the client does not resolve their issue, we go a step further and block it. We already saw a lot of tools waste a lot of resources, including bandwidth, CPU cycles, and energy.\nBlocking outdated clients This website was made with the \u0026ldquo;open\u0026rdquo; mindset of the web. Almost any client should be able to connect, as long as it is fairly modern. For that reason, we don\u0026rsquo;t use the most strict protocol set. This way somewhat older systems can still connect. At the same time, clients that are outdated or simply don\u0026rsquo;t support basic protocols or technologies, will be blocked. For example, this website will only allow HTTPS connections. Older TLS versions are disabled, as well as older ciphers. Does your client not support at least HTTP 1.1 and content encoding, chances are big that it won\u0026rsquo;t be allowed. These older clients should be updated, so they support the modern web. The benefit is also that less data is required to retrieve content.\nWebP and SVG for images Before we used mostly files in JPEG or PNG format. With WebP being introduced in 2010 and available for some time now, most modern browsers support it. Screenshots and other images were converted to WebP and are the preferred resource. Only if the browser does not support it yet, we provide a fallback to PNG or JPEG. Icons and our logo are now Scalable Vector Graphics (SVG), which provides a much sharper result and a reduced file size.\nMore green energy in the mix With the development of the blog, we also looked at how we can increase the amount of green energy. This is energy from renewable sources, including our own solar panels. That is also why we moved some tasks to during the afternoon, when the most solar energy is available, such as some backup tasks.\nReducing data storage Every website will quickly grow when it comes to storage on the disk or in a database. Although we no longer use a database engine, the storage is still a thing. For that reason, we simplified our backup strategy, including the usage of snapshots. Now we can go back longer in time, while reducing the overall storage needs. Also, the backup structure has been simplified, as everything is now in one directory together. With WordPress this was a combination including the WordPress software, plugins, templates, images on the disk, and the database export. With a much simpler disk structure, it is easier to replicate data or make a standalone copy. For example, for disaster recovery we still want to have multiple types of backups and in different locations.\nDuplicate data reduction The backups above are the only exception now when it comes to duplicate data. While building up our internal website structure, we removed as much as possible to limit data duplicates. So we don\u0026rsquo;t have a development, test, and production environment for the website. One system does the development and testing, the output (generated by Hugo) is then synced to production system. Obviously with an optimized method to reduce time, bandwidth, and CPU cycles.\nUser Experience Design (UX) We want the best possible experience when browsing our website, so we applied multiple measures to improve the blog and its interface.\nUser interface Focus on performance Reduce errors in HTML and CSS Limit JavaScript usage Error handling User interface When redesigning the blog, we applied the Mobile First approach. Articles should be readable (and fast) on a mobile phone, so all clutter was removed. A simple header, content area, and footer is what remains. On larger screens, the content area will be capped to ensure that the article is still easy to read.\nHTML and CSS Using HyperText Markup Language (HTML) and Cascading Style Sheets (CSS) most of the pages are formatted and styled. This blog uses HTML5 and we tried to minimize errors in the syntax. This way it should show correctly on as many devices as possible.\nLimited use of JavaScript JavaScript allows developers to use powerful events within a website. Our focus is on getting articles clearly on your screen without JavaScript when possible. The only exception is our search engine, that runs on the client system itself. It requires some logic processing that by default HTML does not support. For that single page we tried to find a small library that allows you to search in all articles, without any overhead on our systems.\nPerformance of the pages Using the proper HTML and CSS syntax is a good start, but not enough. Keeping it as small as possible decreases the overhead on the client. After all, there is less to process, resulting in pages that load and show quicker. So where we can, we stripped out useless HTML tags and don\u0026rsquo;t use a CSS framework. Instead, we only add CSS when needed for proper display. When adding new pieces, we look directly on how to consolidate and decrease overhead. In this case there is additional value in it, as all CSS is available in a \u0026lt;style\u0026gt; tag. The browser does not have to do an extra HTTP request, but can directly start processing the page. This helps with speeding up the page rendering. For those readers active in web design, magic words like Above the Fold, First Paint, First Contentful Paint (FCP), and Largest Contentful Paint (LCP). In simplified terms that would mean that the shorter the time it takes to achieve these, the quicker a website loads and shows on your screen.\nError handling If a page does not exist (404), we point you to the search engine on the website and a link to main page. This way you can easily navigate to a point that works and might help to find you the information that you were looking for.\nFor search engines we started to respond to outdated pages or assets with HTTP 410 (Gone). This way we want to signal them that they should not retry, and instead focus on pages that do exist.\nQuestions or suggestions? Do you have any feedback for our initiative to make this blog more eco-friendly? Let us know!\n","permalink":"https://linux-audit.com/sustainable-web-design/","tags":["website"],"title":"Sustainable web design"},{"categories":["Linux","nginx","Web"],"contents":"Why block POST requests in the first place? Some websites or resources don\u0026rsquo;t need POST requests, such as a statically generated website. It looks like POST requests also take some CPU time within nginx to process them compared with static files. This becomes visible when using the $request_time variable to customize the access log.\n2024-04-02T10:14:39+00:00 404 a.b.c.d \u0026ldquo;POST /xmlrpc.php HTTP/1.1\u0026rdquo; 562 \u0026ldquo;-\u0026rdquo; \u0026ldquo;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36\u0026rdquo; TLSv1.2/ECDHE-ECDSA-AES128-GCM-SHA256 0.218 .\nThe request time in the nginx log is the time between the first byte from the client till the last part of the body being returned to the client. When comparing the request time (0.218) with normal requests, it becomes more obvious. They are almost all \u0026lsquo;0.000\u0026rsquo;, as they can be retrieved from the disk and without much overhead delivered to the client. Especially if you pre-compile files with Brotli or Gzip.\nOptions to filter out POST requests Use an if-statement and $request_method The first option is to use if and filter on the $request_method itself. This statement can be included in the server definition of your virtual host.\nif ($request_method = POST ) { return 405; } This small piece of logic tells nginx that we specifically want to look for POST requests and take an action. In other words, it is like a block list where you define what you don\u0026rsquo;t want. Another way is reversing this and define an allow list instead.\nif ($request_method !~ ^(GET|HEAD)$ ) { return 405; } If the methods are GET or HEAD, then we continue, otherwise we return the HTTP 405 status.\nAn import note about this 405 has to be made. The HTTP spec says that when returning a 405, you need to specify what methods you actually do accept using the Allow header.\nThe actual set of allowed methods is defined by the origin server at the time of each request. An origin server MUST generate an Allow header field in a 405 (Method Not Allowed) response and MAY do so in any other response. An empty Allow field value indicates that the resource allows no methods, which might occur in a 405 response if the resource has been temporarily disabled by configuration.\nThere are a few ways to define this Allow header, with the first being to use add_header.\nadd_header Allow \u0026quot;GET, HEAD\u0026quot; always;\nMost likely your configuration already defines some headers and adding one more is just a small thing. At the same time it is overhead for all legitimate requests. To limit this overhead, we can only show the header for HTTP 405 responses. In your server context (virtual host configuration) you define error_page 405 and tell what it should do. Instead of pointing it to an HTML file, we link it to a location. There we define that it should insert the header and define the allowed methods.\nerror_page 405 @error405; location @error405 { # Insert Allow header to comply with HTTP standard add_header Allow \u0026#34;GET, HEAD\u0026#34; always; } Using an if-statement to process the incoming requests works, but there are alternatives, such as using a map. Let\u0026rsquo;s have a look at that option as well.\nCreating a map and $request_method The second option is creating a map that contains the request methods that we want to block, while allowing the remaining ones that are not specified. Using an if statement we then can define an action based on the request method.\n# Block specific types of request methods (1=block) map $request_method $is_blocked_method { default 0; POST 1; } server { ... # Define the error 405 handler error_page 405 @error405; # Block unwanted methods if ($is_blocked_method) { return 405; break; } location / { try_files $uri $uri/ =404; } location @error405 { # Insert Allow header to comply with HTTP standard add_header Allow \u0026#34;GET, HEAD\u0026#34; always; } ... } Note: map requests can be added to the http context, not to server. So define it above the definition of the virtual host.\nUsing the nginx limit_except option Nginx has the option limit_except which defines what methods are accepted without any restrictions. This is a great way to filter out requests if you only want to allow one type of requests, like GET. It means anything else will be blocked, such as POST requests. Good to know is that when you define GET it will also inherit HEAD. This way you won\u0026rsquo;t break this common method.\nserver { ... location / { # Limit everything NOT being GET (and HEAD), e.g. POST is not allowed # To allow some systems doing a POST, define the \u0026#39;allow\u0026#39; limit_except GET { # allow 1.2.3.4; deny all; } try_files $uri $uri/ =404; } ... } This method takes only three lines! Instead of a 405, it returns a 403. Therefore, no need for adding an Allow header.\nApplying the configuration After making the changes, test your configuration\n# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Then reload your configuration\nservice nginx reload\nWith this action we made your configuration a bit more secure again. What is the next step to further harden your nginx configuration?\n","permalink":"https://linux-audit.com/web/how-to-block-post-requests-in-nginx/","tags":["nginx","web server"],"title":"How to block POST requests in nginx"},{"categories":["SSH"],"contents":"SSH or Secure Shell is a popular protocol for doing system administration on Linux systems. Sometimes you may need to know what version you are running to know if some specific configuration options are available. In this article we have a look at the available options.\nLocal OpenSSH version The easiest way to find the installed OpenSSH version is using the ssh -V command. This works when being logged in to the system itself.\n# ssh -V OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022 So this system is running the 8.9p1 version of OpenSSH.\nQuery installed package version Another option is by looking at the package version itself, as that typically gives a good hint about the version as well.\n# dpkg -l openssh* | grep ssh ii openssh-client 1:8.9p1-3ubuntu0.6 amd64 secure shell (SSH) client, for secure access to remote machines ii openssh-server 1:8.9p1-3ubuntu0.6 amd64 secure shell (SSH) server, for secure access from remote machines ii openssh-sftp-server 1:8.9p1-3ubuntu0.6 amd64 secure shell (SSH) sftp server module, for SFTP access from remote machines Note: the additional grep is used to strip out the headers from the output and make the output less noisy\nLinux distribution Command Alma Linux rpm -qa openssh* Debian dpkg -l openssh* RHEL rpm -qa openssh* Ubuntu dpkg -l openssh* Using another distribution in this list and know the command? Let it [know](/contact/!\nRemote version of OpenSSH server The version of the server might be different than locally installed. The version can easily been seen by connecting to the other system and use the verbose mode option -v.\n# ssh -v localhost OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: Connecting to localhost [127.0.0.1] port 22. debug1: Connection established. debug1: identity file /home/michael/.ssh/id_rsa type -1 debug1: identity file /home/michael/.ssh/id_rsa-cert type -1 debug1: identity file /home/michael/.ssh/id_ecdsa type -1 debug1: identity file /home/michael/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/michael/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/michael/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/michael/.ssh/id_ed25519 type -1 debug1: identity file /home/michael/.ssh/id_ed25519-cert type -1 debug1: identity file /home/michael/.ssh/id_ed25519_sk type -1 debug1: identity file /home/michael/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/michael/.ssh/id_xmss type -1 debug1: identity file /home/michael/.ssh/id_xmss-cert type -1 debug1: identity file /home/michael/.ssh/id_dsa type -1 debug1: identity file /home/michael/.ssh/id_dsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 pat OpenSSH* compat 0x04000000 Note: replace localhost with the IP address or hostname of the server\nDo you know about an alternative method to find details about the OpenSSH version?\n","permalink":"https://linux-audit.com/how-to-find-openssh-version/","tags":["howto","server-security","ssh","ssh_config"],"title":"How to find the OpenSSH version"},{"categories":["System Administration"],"contents":"Rnr The first tool to cover is called rnr and is written in Rust. It can be downloaded on GitHub where also some good examples can be found on how to use the tool.\nLet\u0026rsquo;s try it out on a directory that we have with Markdown files. Due to a conversion, the file names include a date. As this is no longer needed, we want to strip out the date and only get the bit after the third hyphen.\nOriginal file name: 2014-10-25-how-to-create-custom-tests-in-lynis.md Wanted file name: how-to-create-custom-tests-in-lynis.md # rnr --dry-run \u0026#39;\\d+-\\d+-\\d+-\u0026#39; \u0026#39;\u0026#39; ./*.md This is a DRY-RUN ./2014-10-25-how-to-create-custom-tests-in-lynis.md -\u0026gt; ./how-to-create-custom-tests-in-lynis.md ./2016-01-05-how-to-determine-a-file-type-on-linux.md -\u0026gt; ./how-to-determine-a-file-type-on-linux.md ./2015-03-30-viewing-available-test-categories-in-lynis.md -\u0026gt; ./viewing-available-test-categories-in-lynis.md ./2014-11-24-auditing-systemd-solving-failed-units-with-systemctl.md -\u0026gt; ./auditing-systemd-solving-failed-units-with-systemctl.md Parameter Explanation --dry-run Don\u0026rsquo;t make changes right away, but show what will be done '(\\d+)-(\\d+)-(\\d+)-' Capture three occurrences of just numbers '' Replace with empty string, as we want to strip the date ./*.md Only apply this to the Markdown files in this current directory The nice part of rnr is that it shows the rename it will do. As this looks good, we can tell it to perform the action. In that case replace the dry-run with --force to make the actual changes.\nRename tool The rename tool with the similar name rename can be found on many Linux systems. It used a Perl syntax for the matching, so with some fiddling we can do the same bulk action as before.\nIf the tool is not installed yet, here are some pointers on the right package\nLinux distribution Package name Installation Ubuntu rename apt install rename Usage # rename --nono \u0026#39;s/\\d{4}-\\d{2}-\\d{2}-//\u0026#39; ./*.md rename(./2014-03-02-lynis-stuck-during-testing.md, ./lynis-stuck-during-testing.md) rename(./2014-03-06-linux-audit-auditing-network-configuration.md, ./linux-audit-auditing-network-configuration.md) rename(./2014-03-09-securing-linux-audit-lynis.md, ./securing-linux-audit-lynis.md) Parameter Explanation --nono Do not make actual changes 's/\\d{4}-\\d+{2}-\\d{2}-//' Do a search (s), first 4 digits, hyphen, 2 digits, hyphen, 2 digits, hyphen. No replacement text, as we want to strip text from the file name regex-rename Another great utility can be found in the Python Package Index (PyPi) and is named regex-rename. As the name implies, it also support regular expressions.\nInstallation pip3 install regex-rename\nTip: do the installation in a virtual environment (python3-virtualenv)\nUsage The next step is forming the regular expression. This utility does a \u0026ldquo;dry-run\u0026rdquo; by default and requires the --rename option to make actual changes.\n# regex-rename \u0026#34;(\\d{4})-(\\d{2})-(\\d{2})-(.*)\u0026#34; \u0026#34;\\4\u0026#34; [2024-03-28 14:03:51] DEBUG matching regular expression pattern to files: pattern=(\\d{4})-(\\d{2})-(\\d{2})-(.*) replacement=\\4 dry_run=True full_match=False recursive=False padding=None [2024-03-28 14:03:51] INFO matched file: from=2014-03-02-lynis-stuck-during-testing.md to=lynis-stuck-during-testing.md group_1=2014 group_2=03 group_3=02 group_4=lynis-stuck-during-testing.md [2024-03-28 14:03:51] INFO matched file: from=2014-03-06-linux-audit-auditing-network-configuration.md to=linux-audit-auditing-network-configuration.md group_1=2014 group_2=03 group_3=06 group_4=linux-audit-auditing-network-configuration.md [2024-03-28 14:03:51] INFO matched file: from=2014-03-09-securing-linux-audit-lynis.md to=securing-linux-audit-lynis.md group_1=2014 group_2=03 group_3=09 group_4=securing-linux-audit-lynis.md [2024-03-28 14:03:51] INFO files matched the pattern: matched=3 mismatched=0 Let\u0026rsquo;s have a look at the regular expression first.\nRegular expression part Explanation \u0026quot; Open the regular expression (\\d{4})- Match 4 digits, followed by a hyphen (\\d{2})-(\\d{2})- Match 2 digits, hyphen, 2 digits, hyphen (.*) Match all characters, so that we can use this part as our replacement \u0026quot; Close the regular expression The output of the tool is colored and shows what it is trying to do. We see also the dry_run=True that tells us that it won\u0026rsquo;t make any changes. It shows what files match and what the new file name would be. Another great tool do do a batch rename action.\nGot another tool that you really like to use for batch renaming? Let us know!\n","permalink":"https://linux-audit.com/linux-tools-to-bulk-rename-files/","tags":["file system","howto","linux"],"title":"Linux tools to bulk rename files"},{"categories":["Authentication","Passwords"],"contents":"Sometimes you might want to check if an account on the system has a password set. One of the reasons is to disable those, so you can enforce that only SSH authentication might be used, for example.\nThere are a few ways to see if a password is set.\nUsing the passwd command The first command that comes to mind is using the passwd command. Normally you would use that to change your password, but it can actually also reveal useful details about existing accounts. Using the -S option we can request such details.\npasswd -S michael\nThe output might look something like this.\nmichael P 01/13/2024 0 99999 7 -1\nSo what does this single line output mean?\nField Explanation michael Username P P indicates an usable password is set, L means it is locked 01/13/2024 Last password change 0 Minimum age (in days) 99999 Maximum age (in days) 7 Warning period (in days) -1 Inactivity period for the password So on the account above you can see the password was set in January 2024 and it has an usable password.\nUsing getent shadow Linux uses different types of databases to store information, including those related to name resolution and authentication. One way is to look in the related files /etc/passwd or /etc/shadow. An easier way is to query the related shadow database directly using the getent command.\n# getent shadow michael michael:$6$xyz$VKswtvLoVpOLcpjDMIFXhxa8ukqqKSKHjcPBLZUk9NxWldmlFQY4stUGo.QjEhav7mp86ih2PRqYPqjkhWi5y.:19735:0:99999:7::: # getent shadow www-data www-data:*:19579:0:99999:7::: In this output we clearly see that the first account has a long string characters in the second field, while the second one has not. This long string of characters is the stored password.\nField Explanation $6 Refers to the SHA512 cryptographic hash function xyz salt for the hashing function VKis\u0026hellip;5y. Password in hashed format Using chage The chage command is great to determine the password policy by using the -l option. Unfortunately, it not as trustworthy to find out if a password has been set. Let\u0026rsquo;s have a look at the following output.\nchage -l www-data Last password change\t: Aug 10, 2023 Password expires\t: never Password inactive\t: never Account expires\t: never Minimum number of days between password change\t: 0 Maximum number of days between password change\t: 99999 Number of days of warning before password expires\t: 7 This would indicate that a password is set, as it has been changed, right? When we look at the passwd -S output, we see that is is locked though.\npasswd -S www-data www-data L 08/10/2023 0 99999 7 -1 So this account has no password set, even though the field Last password change would give the impression is has.\nFound another method to see if an account has a password, or got a suggestion? Love to hear!\n","permalink":"https://linux-audit.com/how-to-check-if-an-account-has-a-password-set/","tags":["authentication","howto","password","passwd"],"title":"How to test if an account has a password set?"},{"categories":["Web"],"contents":"After migrating this blog to Hugo we performed some optimization steps to ensure it is as quick as possible. Pages are slim and small in size, but still can be compressed. Normally we would do this on the end of the web server, by enabling dynamic compression. So each time a client requested a compressed page, the web server would compress is and send over the data. This time we turned things around.\nCompress data upfront Using the brotli command we can compress files and store them on the webserver. So if we have a file named index.html, we can tell the tool to compress it. By using the --best parameter, we tell it to get the best possible compression. After all, it\u0026rsquo;s a one-time action, so why not take our best shot?\nUsing the tool is very straightforward: brotli --best index.html\nThe brotli tool is usually not installed by default\nUsing Firefox to test if compression is working Open Firefox Press CTRL + SHIFT + i Click on the Network tab Open the web page to test, hard refresh if needed (CTRL + SHIFT + r) Look at the first column to see if a 200 response code is returned Look at the columns Transferred and Size to see if they are different Click on the request, which opens up all details Confirm that the browser supports compression. Most likely the Request Headers section will show Accept-Encoding: gzip, deflate, br to indicate that it supports multiple types of compression Then confirm that Response Headers show content-encoding. If Brotli is used, then the value will display this as br Using curl to test if compression is working To test if compression is working as expected, we can also use the cURL utility. Nowadays it is installed on most systems, so why not use that instead?\ncurl --head --header 'Accept-Encoding: br' --silent https://linux-audit.com/ | grep content-encoding\nSo what does it do?\nParameter Explanation --head performs a HEAD request and returns only the headers, not the actual data --header includes the header where we tell that we would accept Brotli encoding --silent suppresses the progress output within cURL If everything is working as expected, the output should include the encoding provided by the web server.\ncontent-encoding: br\nGood luck in further optimizing your web server(s)!\n","permalink":"https://linux-audit.com/how-to-test-if-a-website-supports-brotli-or-gzip/","tags":["brotli","curl","gzip","howto","web","web server"],"title":"How to test if a website supports Brotli or Gzip compression"},{"categories":["SSH"],"contents":"Rsync is still one of the most popular tools to synchronize files between two systems. Although it has a few caveats when dealing with special files, it can do its job very well. In this explainer we will show how to use it in combination with SSH and at the same restrict SSH access to only allow the rsync job to run.\nIn this article we refer to system01 having the original files and it wants to send them to the receiving system (system02)\nCreate user on receiving system The system that receives the files (system02) should have a user that will be used for the file transport. Typically this is a dedicated user for file transfers. For this example we call it backupuser. The user does not need a password, as we don\u0026rsquo;t want interactive logins.\nadduser --disabled-password --shell /bin/bash --gecos \u0026quot;Backup user\u0026quot; backupuser\nGenerate the key Using the ssh-keygen utility we can create a new key. In this example we will store the SSH keys in /data/ssh-keys and restrict access, so let\u0026rsquo;s create that path first.\nmkdir -p /data/ssh-keys chmod 700 /data/ssh-keys Next step is the creating of the key.\nssh-keygen -t ed25519 -f /data/ssh-keys/backupuser-key -C \u0026quot;backupuser for system1\u0026quot;\nThe -t defines the type of key, in this case Ed25519. For modern versions of SSH this will be the default, but older systems might still use RSA. By defining the type we ensure that we have the right type.\nAdd entry to authorized_keys On the receiving system (system02) we need to create a file .ssh/authorized_keys. Create the related paths and set the permissions.\ncd /home/backupuser mkdir .ssh chmod 700 .ssh touch authorized_keys chmod 600 authorized_keys Insight: SSH is very restrictive when it comes to file permissions. If they are too loose, the usage of SSH keys won\u0026rsquo;t work.\nSet permissions mkdir /data/backups/system01 chown backupuser:backupuser /data/backups/system01 chmod 750 /data/backups/system01 Copy the restricted rsync access to the authorized_keys Within the authorized_keys file we now need to tell what command can be executed when we use a particular key.\ncommand=\u0026ldquo;rsync \u0026ndash;server -vlogDtprCze.iLsfx \u0026ndash;delete . /data/backups/system01\u0026rdquo;,no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-ed25519 AAAA\u0026hellip;\u0026hellip;.. backupuser for system1\nThis line tells SSH that it should restrict access (e.g. no shell, no forwarding of anything related to the SSH agent or X11). It also defines what command is allowed, which is restricted to the rsync command.\nImportant is that we need to replace the last part, as that includes the public key and key comment. This part can be found on the sending system in /data/ssh-keys/backupuser-key.pub.\nInitiate the synchronization On system01 we will now run the action to start the copy.\nrsync -e 'ssh -i rsync_key' --archive --cvs-exclude --checksum --compress --verbose --delete /path/to/files/to/backup backupuser@system02:/data/backups/system01\nTip: add the first time the \u0026ndash;dry-run parameter so that you can see and test that everything should work as expected\nIf everything works as expected, the files should be synced now to the other system.\n","permalink":"https://linux-audit.com/ssh/restrict-ssh-access-to-only-allow-rsync/","tags":["authorized_keys","ssh","ssh-keygen"],"title":"Restrict SSH access to only allow rsync"},{"categories":null,"contents":"This page contains the most recent changes to the blog.\n","permalink":"https://linux-audit.com/website/latest-changes/","tags":["website"],"title":"Latest changes"},{"categories":["Lynis"],"contents":"After almost a year of work, we are excited to share news about the major 3.x release! It is major for multiple reasons, including the number of submissions from the community and some breaking changes. Some core functions have been rewritten and several new functions were added.\nAnother important area for this release is security. Being a security tool, we want Lynis to be as safe as possible, even though shell script is not specifically known for that. So this subject received additional focus on that as well.\nSpecial thanks Before talking about all new features and changes, let\u0026rsquo;s first thank a few individuals who contributed in a special way to this release. Their work shaped not only this release but also the future of Lynis.\nSander Bos discovered several security issues on how Lynis deals with the input and output of data. Based on his work, we requested several CVEs and patched the related areas. Furthermore, it inspired us to apply additional security measures to prevent similar issues. Thanks Sander, for your great insights, patience, and work!\nAnother person that deserves a big thanks is Topi Miettinen. His recent code contributions are of good quality, making it easy for the Lynis project to review and merge them. Most of his contributed work is now part of this major release. Thanks!\nBreaking changes So this new Lynis release might also break a few things. Let\u0026rsquo;s have a detailed look at them. This way you can determine what is applicable to your situation and resolve them.\nProfiles Especially when it comes to parsing your custom profile. If you are using the old-style notation, then the following message might show up:\n[WARNING]: Your profile contains old-style configuration entries. See log file for more details and how to convert these entries If you receive this message, then most likely you have to adjust your custom.prf. Look in the default profile (default.prf) to see the correct notation of a particular entry.\nBackground: In early versions, Lynis used a colon to split a key/value pair and underscores (e.g. is_enabled:1). In newer versions, we moved to a value assignment by using the equals (=) sign and dashes (e.g. is-enabled=1). Readability and simplicity was the rationale to make this change.\nCommand line options Some of the options (like -c) are there from the very beginning. Since Lynis 2.x these vague options were rewritten to more instructive commands (-c became audit system). Vague command-line options have no place in easy-to-use software. With Lynis 3.x we decided to start removing them. To encourage users that still use the old switches, we will show when an option was deprecated and what switch or command has replaced it.\nThis option (-c) is deprecated. Use: lynis audit system [options] Tip: check your cron job. Are you still using an old command-line switch?\nSecurity Let\u0026rsquo;s talk about security! This major release comes with additional safeguards to make Lynis safer to use and harder to misuse. To better understand the new and enhanced security measures, it may be a good thing to look at the discovered issues first.\nVulnerabilities With the help of some skilled individuals, we found some weaknesses in how Lynis parses data returned by common system tools. It made some assumptions that the output could be trusted. This approach is fine for most systems and tools, as data is correctly filtered or have no special characters in it. But if a system does host malicious users, things might change. One issue was discovered related to the log and report file. Another issue is information disclosure. Let\u0026rsquo;s have a look at the log file and report file first.\nVulnerability: dealing with log and report data When Lynis performs its scan, it may write detailed information in its log file. This data is useful for learning what specifically was tested and discovered. Then there is useful data about the audit itself and stored in the report file (lynis-report.dat). Think data such as OS information, installed software packages, etc.\nMost of the data stored in the log file and report file is simple text. If one is to manipulate that, then this still does not really introduce a vulnerability. The exception is when some of this data is parsed later and reused. That is exactly was happens near the end of the audit and in some specific tests. So if an attacker can manipulate the log file or report file, it may result in unexpected behavior along the way.\nWith the detailed help of Sander Bos, a weak spot was discovered when running Lynis as a non-privileged user. Even with some checks in place, it was possible to circumvent the safety checks. This makes it possible for systems that have no relevant security measure, to hijack the log and report files by setting up a symlink. Linux systems have since kernel 3.6 a way to guard against these symlink attacks. The relevant sysctl keys are fs.protected_hardlinks and fs.protected_symlinks. So the specific attack probably won\u0026rsquo;t work on most systems, but in the end it is still a vulnerability.\nTo resolve any issues with hijacking files and possibly malicious data, we decided to no longer store the log file and report /tmp. Instead, when a non-privileged user runs Lynis, it will go to the personal home directory. For the root user (or privileged), it will still go to /var/log.\nVulnerability: information disclosure Another issue that was discovered by Sander Bos is related to leaking the license key. During the data upload to a central system, there is a short window of time in which the license key can be discovered from the process table. The impact of the information disclosure is limited, as it won\u0026rsquo;t provide any access to existing accounts or data. Knowing the license key could be abused to flood a system with incorrect or useless data, or exhaust the maximum number of uploads per day.\nNote: Most Lynis users will not have the license key configured, as this is limited to those using Lynis Enterprise. If you are using Linux, you can also use the hidepid option on /proc to prevent this information disclosure.\nSecurity measure: additional input/output filtering To counter possible unexpected behavior when running Lynis and external commands, we added several security safe-guards. One of them is to filter unexpected data, whenever it comes from profiles, the report, or the output of commands.\nAccept Known Good Where possible, we limit the data to characters that we expect to be returned, the so-called Accept-Known-Good approach. For example, most of the values defined in the profile could be limited to a regular expression containing a fixed set (e.g. a-z, A-Z, 0-9, and a hyphen). When unexpected characters are discovered during the parsing of the profile, program execution will stop. Lynis will quit and notify the user.\nReject known bad Sometimes we can\u0026rsquo;t be sure about the input or output. This is especially challenging with provided program parameters or output of external tools. The user could provide unexpected characters, tools may crash and return garbage. If we don\u0026rsquo;t know what to expect, we use the Reject-Known-Bad approach. For example, filtering out escape characters or limit output to just a single line. Another option is to have the od command convert the output, before we ingest it. Nothing is fool-proof, but every measure in this layered security approach may reduce the risk.\nFunction: SafeInput Besides performing input and output validation, we want to keep the code as simple as possible. This is especially needed when there is a repeat similar actions, such as calling for an external tool and parse the output. To help with this, the function SafeInput uses the described approaches above.\nSecurity measure: $PATH Most users will have a common set of directories listed in their PATH variable, such as /bin, /sbin, /usr/bin, etc. Lynis uses this list to determine where to look for binaries. As this is also some form of input, additional checks have been added in this area as well. Because we want to trust every single user, but who knows what odd things people might have in there, by accident or on purpose\u0026hellip;\nNew tests This major release also extends the existing set of tests by at least 20, and includes are variety of areas. Although too much to name, it was especially the cryptography group that got several new tests, such as tests for disk encryption and the use of entropy enhancing generators.\nTip: you can easily write your own tests, as Lynis is written in shell script. Create the file tests_custom in the includedir (run: lynis show includedir). Use the tests_custom.template file to see some examples, or look into the existing tests on how they work.\nOperating systems and end-of-life detection Another area that received some attention is the OS detection routine itself. Especially with more Linux distributions that switched to systemd as its service manager (and what not\u0026hellip;), some of the OS detection was simplified and more structured.\nImproved OS support and functions Lynis was originally developed on a FreeBSD system and supported a wide range of operating systems at its first release. We are proud that Lynis is still a lightweight tool and yet supports so many platforms. At the same time, some generic tests or functions might fail. This major release holds several patches to improve OS detection and execution tests and functions.\nEOL detection While we understand that not everyone can run the latest and greatest version of an operating system, there are still many companies using end-of-life systems. Although we can\u0026rsquo;t change that, we can at least flag them and inform the user. Maybe it helps to convince management that another system imposes extra risk and additional measures are needed.\nSince Lynis 2.7.2 a function was added to do the end-of-life check. The operating system together with the end-of-life timestamp is compared against the current date and time. Did you go over the end-of-life? Then a message will be displayed. A manual check can be done by using lynis show eol. This major release improved the end-of-life check and several entries were added with the help of the community.\nMigration tips from 2.x to 3.x We suggest all users migrating as early as possible to the new 3.x release. The main reason is that there will be no new 2.x releases and all new changes will be part of the 3.x branch. Depending on your operating system and software repositories that are used, the switch may be as simple as doing a package upgrade. However, we do suggest you do perform some basic testing first.\nTest the upgrade If you installed Lynis as a package, you can fairly easily test if your configuration keeps working after upgrading. Here is one approach:\nStep 1: deinstall your Lynis package (e.g. apt remove lynis), but leave the existing configuration files Step 2: download Lynis from GitHub (using git clone or download the tarball) Step 3: run lynis show profiles Step 4: got an error on this command? See the tips in this post Step 5: no error? Great, confirm that your custom.prf (e.g. /etc/lynis/custom.prf) shows up in the output Step 6: run lynis system audit Step 7: check for any issues (old configuration entries, unexpected characters, unexpected screen output) Step 8: remove the new 3.x version and clean up the directory Step 9: install the 2.x version again (apt install lynis) Step 10: check your cronjob and replace old command flags in advance (e.g. replace -c to audit system) By performing these steps should be able to determine if something breaks when switching all your systems to the new 3.x release. Most likely the impact will be limited, but better safe than sorry.\nDon\u0026rsquo;t copy the default profile! Did you make a copy of the default profile? This is the time to start fresh. As shown in the default profile, only copy the relevant entries that you want to change into your custom.prf. The default profile is always loaded and sets the default values. The custom profile builds upon these defaults and defines your specific preferences. So keep that custom profile small and readable!\nWhat is next? The first release of Lynis was in 2007. Even though it is already 13 years ago Lynis is still relevant, with new users every day. Therefore this major release is the beginning of a new era of the tool. Let\u0026rsquo;s have a look at the future.\nMore frequent releases again! Now that we finished this major release, the time between two releases will be much shorter again. You can expect one or two releases per month like we used to do. The project remains healthy and having regular releases shows both users and contributors that changes are implemented.\nThe focus will be on completing several aspects on the side-line of the project, such as the website. One area that we like to improve is documentation. Not everyone is a native English speaker, so we like to refresh all documentation pages and make them available in more different languages.\nTo get the documentation translated, we will ask our community to chime in, as we did in the past for translation several parts of Lynis itself. Speaking about that, some of the output is still English-only. So that is something else we like to cover.\nWant to help? The beauty of open-source software is that it allows many people to use the software. During the last years, we have seen a great increase in contributions, from translations to code enhancements. At the same time it is completely understandable that depending on your time and skill set, your options to help are limited. Events like the COVID-19 pandemic will put even more pressure on all of us. Still, there are few smalls things you can do that may result in a great impact. Here are some ideas:\nSend out a tweet about your favorite tool(s) and what you like about it Have a look at the available translations and see if you can help there Read the documentation carefully, point out areas that can be improved, such as grammar issues Reach out to your favorite projects and send a small \u0026ldquo;Thank you\u0026rdquo; Share this blog post with friends or colleagues Enjoy the new release I hope you will like the new Lynis release. Give it a try, and let me know how you like it. Stay safe!\n","permalink":"https://linux-audit.com/lynis/lynis-3-major-release/","tags":["auditing","lynis","system hardening"],"title":"Major release: Lynis 3.x"},{"categories":["Development"],"contents":"Some of the true craftsmanship in the world we take for granted. One of these things is the common tools on Linux, like ps and ls. Even though the commands might be perceived as simple, there is more to it when looking under the hood. This is where ELF or the Executable and Linkable Format comes in. A file format that used a lot, yet truly understood by only a few. Let\u0026rsquo;s get this understanding with this introduction tutorial!\nBy reading this guide, you will learn:\nWhy ELF is used and for what kind of files Understand the structure of ELF and the details of the format How to read and analyze an ELF file such as a binary Which tools can be used for binary analysis What is an ELF file? ELF is the abbreviation for Executable and Linkable Format and defines the structure for binaries, libraries, and core files. The formal specification allows the operating system to interpreter its underlying machine instructions correctly. ELF files are typically the output of a compiler or linker and are a binary format. With the right tools, such file can be analyzed and better understood.\nWhy learn the details of ELF? Before diving into the more technical details, it might be good to explain why an understanding of the ELF format is useful. As a starter, it helps to learn the inner workings of our operating system. When something goes wrong, we might better understand what happened (or why). Then there is the value of being able to research ELF files, especially after a security breach or discover suspicious files. Last but not least, for a better understanding while developing. Even if you program in a high-level language like Golang, you still might benefit from knowing what happens behind the scenes.\nSo why learn more about ELF?\nGeneric understanding of how an operating system works Development of software Digital Forensics and Incident Response (DFIR) Malware research (binary analysis) From source to process So whatever operating system we run, it needs to translate common functions to the language of the CPU, also known as machine code. A function could be something basic like opening a file on disk or showing something on the screen. Instead of talking directly to the CPU, we use a programming language, using internal functions. A compiler then translates these functions into object code. This object code is then linked into a full program, by using a linker tool. The result is a binary file, which then can be executed on that specific platform and CPU type.\nBefore you start This blog post will share a lot of commands. Don\u0026rsquo;t run them on production systems. Better do it on a test machine. If you like to test commands, copy an existing binary and use that. Additionally, we have provided a small C program, which can you compile. After all, trying out is the best way to learn and compare results.\nThe anatomy of an ELF file A common misconception is that ELF files are just for binaries or executables. We already have seen they can be used for partial pieces (object code). Another example is shared libraries or even core dumps (those core or a.out files). The ELF specification is also used on Linux for the kernel itself and Linux kernel modules.\nThe file command shows some basics about this binary file\nStructure Due to the extensible design of ELF files, the structure differs per file. An ELF file consists of:\nELF header File data With the readelf command, we can look at the structure of a file and it will look something like this:\nDetails of an ELF binary\nELF header As can be seen in this screenshot, the ELF header starts with some magic. This ELF header magic provides information about the file. The first four hexadecimal parts define that this is an ELF file (45=E,4c=L,46=F), prefixed with the 7f value.\nThis ELF header is mandatory. It ensures that data is correctly interpreted during linking or execution. To better understand the inner working of an ELF file, it is useful to know this header information is used.\nClass After the ELF type declaration, there is a Class field defined. This value determines the architecture for the file. It can a 32-bit (=01) or 64-bit (=02) architecture. The magic shows a 02, which is translated by the readelf command as an ELF64 file. In other words, an ELF file using the 64-bit architecture. Not surprising, as this particular machine contains a modern CPU.\nData Next part is the data field. It knows two options: 01 for LSB Least Significant Bit , also known as little-endian. Then there is the value 02, for MSB (Most Significant Bit, big-endian). This particular value helps to interpret the remaining objects correctly within the file. This is important, as different types of processors deal differently with the incoming instructions and data structures. In this case, LSB is used, which is common for AMD64 type processors.\nThe effect of LSB becomes visible when using hexdump on a binary file. Let\u0026rsquo;s show the ELF header details for /bin/ps.\n$ hexdump -n 16 /bin/ps 0000000 457f 464c 0102 0001 0000 0000 0000 0000 0000010 We can see that the value pairs are different, which is caused by the right interpretation of the byte order.\nVersion Next in line is another \u0026ldquo;01\u0026rdquo; in the magic, which is the version number. Currently, there is only 1 version type: currently, which is the value \u0026ldquo;01\u0026rdquo;. So nothing interesting to remember.\nOS/ABI Each operating system has a big overlap in common functions. In addition, each of them has specific ones, or at least minor differences between them. The definition of the right set is done with an Application Binary Interface (ABI ). This way the operating system and applications both know what to expect and functions are correctly forwarded. These two fields describe what ABI is used and the related version. In this case, the value is 00, which means no specific extension is used. The output shows this as System V .\nABI version When needed, a version for the ABI can be specified.\nMachine We can also find the expected machine type (AMD64) in the header.\nType The type field tells us what the purpose of the file is. There are a few common file types.\nCORE (value 4) DYN (Shared object file), for libraries (value 3) EXEC (Executable file), for binaries (value 2) REL (Relocatable file), before linked into an executable file (value 1) See full header details While some of the fields could already be displayed via the magic value of the readelf output, there is more. For example for what specific processor type the file is. Using hexdump we can see the full ELF header and its values.\n7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 |.ELF............| 02 00 3e 00 01 00 00 00 a8 2b 40 00 00 00 00 00 |..\u0026lt;......+@.....| 40 00 00 00 00 00 00 00 30 65 01 00 00 00 00 00 |@.......0e......| 00 00 00 00 40 00 38 00 09 00 40 00 1c 00 1b 00 |....@.8...@.....| (output created with hexdump -C -n 64 /bin/ps)\nThe highlighted field above is what defines the machine type. The value 3e is 62 in decimal, which equals to AMD64.\nWhile you can do a lot with a hexadecimal dump, it makes sense to let tools do the work for you. The dumpelf tool can be helpful in this regard. It shows a formatted output very similar to the ELF header file. Great to learn what fields are used and their typical values.\nWith all these fields clarified, it is time to look at where the real magic happens and move into the next headers!\nFile data Besides the ELF header, ELF files consist of three parts.\nProgram Headers or Segments (9) Section Headers or Sections (28) Data Before we dive into these headers, it is good to know that ELF has two complementary \u0026ldquo;views\u0026rdquo;. One is to be used for the linker to allow execution (segments). The other one for categorizing instructions and data (sections). So depending on the goal, the related header types are used. Let\u0026rsquo;s start with program headers, which we find on ELF binaries.\nProgram headers An ELF file consists of zero or more segments, and describe how to create a process/memory image for runtime execution. When the kernel sees these segments, it uses them to map them into virtual address space, using the mmap(2) system call. In other words, it converts predefined instructions into a memory image. If your ELF file is a normal binary, it requires these program headers. Otherwise, it simply won\u0026rsquo;t run. It uses these headers, with the underlying data structure, to form a process. This process is similar for shared libraries.\nAn overview of program headers in an ELF binary\nWe see in this example that there are 9 program headers. When looking at it for the first time, it hard to understand what happens here. So let\u0026rsquo;s go into a few details.\nGNU_EH_FRAME\nThis is a sorted queue used by the GNU C compiler (gcc). It stores exception handlers. So when something goes wrong, it can use this area to deal correctly with it.\nGNU_STACK\nThis header is used to store stack information. The stack is a buffer, or scratch place, where items are stored, like local variables. This will occur with LIFO (Last In, First Out), similar to putting boxes on top of each other. When a process function is started a block is reserved. When the function is finished, it will be marked as free again. Now the interesting part is that a stack shouldn\u0026rsquo;t be executable, as this might introduce security vulnerabilities. By manipulation of memory, one could refer to this executable stack and run intended instructions.\nIf the GNU_STACK segment is not available, then usually an executable stack is used. The scanelf and execstack tools are two examples to show the stack details.\n$ scanelf -e /bin/ps TYPE STK/REL/PTL FILE ET_EXEC RW- R-- RW- /bin/ps $ execstack -q /bin/ps - /bin/ps Commands to see program headers\ndumpelf (pax-utils) elfls -S /bin/ps eu-readelf -program-headers /bin/ps ELF sections Section headers The section headers define all the sections in the file. As said, this \u0026ldquo;view\u0026rdquo; is used for linking and relocation.\nSections can be found in an ELF binary after the GNU C compiler transformed C code into assembly, followed by the GNU assembler, which creates objects of it.\nAs the image above shows, a segment can have 0 or more sections. For executable files there are four main sections: .text, .data, .rodata, and .bss. Each of these sections is loaded with different access rights, which can be seen with readelf -S.\n.text Contains executable code. It will be packed into a segment with read and execute access rights. It is only loaded once, as the contents will not change. This can be seen with the objdump utility.\n12 .text 0000a3e9 0000000000402120 0000000000402120 00002120 2**4\nCONTENTS, ALLOC, LOAD, READONLY, CODE\n.data Initialized data, with read/write access rights\n.rodata Initialized data, with read access rights only (=A).\n.bss Uninitialized data, with read/write access rights (=WA)\n[24] .data PROGBITS 00000000006172e0 000172e0 0000000000000100 0000000000000000 **WA** 0 0 8 [25] .bss NOBITS 00000000006173e0 000173e0 0000000000021110 0000000000000000 **WA** 0 0 32 Commands to see section and headers\ndumpelf elfls -p /bin/ps eu-readelf -section-headers /bin/ps readelf -S /bin/ps objdump -h /bin/ps Section groups Some sections can be grouped, as they form a whole, or in other words be a dependency. Newer linkers support this functionality. Still, this is not common to find that often:\n$ readelf -g /bin/ps There are no section groups in this file. While this might not be looking very interesting, it shows a clear benefit of researching the ELF toolkits which are available, for analysis. For this reason, an overview of tools and their primary goal have been included at the end of this article.\nStatic versus Dynamic binaries When dealing with ELF binaries, it is good to know that there are two types and how they are linked. The type is either static or dynamic and refers to the libraries that are used. For optimization purposes, we often see that binaries are \u0026ldquo;dynamic\u0026rdquo;, which means it needs external components to run correctly. Often these external components are normal libraries, which contain common functions, like opening files or creating a network socket. Static binaries, on the other hand, have all libraries included. It makes them bigger, yet more portable (e.g. using them on another system).\nIf you want to check if a file is statically or dynamically compiled, use the file command. If it shows something like:\n$ file /bin/ps /bin/ps: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), **dynamically linked (uses shared libs)**, for GNU/Linux 2.6.24, BuildID[sha1]=2053194ca4ee8754c695f5a7a7cff2fb8fdd297e, stripped To determine what external libraries are being used, simply use the ldd command on the same binary:\n$ ldd /bin/ps linux-vdso.so.1 =\u0026gt; (0x00007ffe5ef0d000) libprocps.so.3 =\u0026gt; /lib/x86_64-linux-gnu/libprocps.so.3 (0x00007f8959711000) libc.so.6 =\u0026gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f895934c000) /lib64/ld-linux-x86-64.so.2 (0x00007f8959935000) Tip: To see underlying dependencies, it might be better to use the lddtree utility instead.\nTools for binary analysis When you want to analyze ELF files, it is definitely useful to look first for the available tooling. Some of the software packages available provide a toolkit to reverse engineer binaries or executable code. If you are new to analyzing ELF malware or firmware, consider learning static analysis first. This means that you inspect files without actually executing them. When you better understand how they work, then move to dynamic analysis. Now you will run the file samples and see their actual behavior when the low-level code is executed as actual processor instructions. Whatever type of analysis you do, make sure to do this on a dedicated system, preferably with strict rules regarding networking. This is especially true when dealing with unknown samples or those are related to malware.\nPopular tools Radare2 The Radare2 toolkit has been created by Sergi Alvarez. The \u0026lsquo;2\u0026rsquo; in the version refers to a full rewrite of the tool compared with the first version. It is nowadays used by many reverse engineers to learn how binaries work. It can be used to dissect firmware, malware, and anything else that looks to be in an executable format.\nSoftware packages Most Linux systems will already have the the binutils package installed. Other packages might help with showing much more details. Having the right toolkit might simplify your work, especially when doing analysis or learning more about ELF files. So we have collected a list of packages and the related utilities in it.\nelfutils /usr/bin/eu-addr2line /usr/bin/eu-ar - alternative to ar, to create, manipulate archive files /usr/bin/eu-elfcmp /usr/bin/eu-elflint - compliance check against gABI and psABI specifications /usr/bin/eu-findtextrel - find text relocations /usr/bin/eu-ld - combining object and archive files /usr/bin/eu-make-debug-archive /usr/bin/eu-nm - display symbols from object/executable files /usr/bin/eu-objdump - show information of object files /usr/bin/eu-ranlib - create index for archives for performance /usr/bin/eu-readelf - human-readable display of ELF files /usr/bin/eu-size - display size of each section (text, data, bss, etc) /usr/bin/eu-stack - show the stack of a running process, or coredump /usr/bin/eu-strings - display textual strings (similar to strings utility) /usr/bin/eu-strip - strip ELF file from symbol tables /usr/bin/eu-unstrip - add symbols and debug information to stripped binary Insight: the elfutils package is a great start, as it contains most utilities to perform analysis.\nelfkickers /usr/bin/ebfc - compiler for Brainfuck programming language /usr/bin/elfls - shows program headers and section headers with flags /usr/bin/elftoc - converts a binary into a C program /usr/bin/infect - tool to inject a dropper, which creates setuid file in /tmp /usr/bin/objres - creates an object from ordinary or binary data /usr/bin/rebind - changes bindings/visibility of symbols in ELF file /usr/bin/sstrip - strips unneeded components from ELF file Insight: the author of the ELFKickers package focuses on manipulation of ELF files, which might be great to learn more when you find malformed ELF binaries.\npax-utils /usr/bin/dumpelf - dump internal ELF structure /usr/bin/lddtree - like ldd, with levels to show dependencies /usr/bin/pspax - list ELF/PaX information about running processes /usr/bin/scanelf - wide range of information, including PaX details /usr/bin/scanmacho - shows details for Mach-O binaries (Mac OS X) /usr/bin/symtree - displays a leveled output for symbols Notes: Several of the utilities in this package can scan recursively in a whole directory. Ideal for mass-analysis of a directory. The focus of the tools is to gather PaX details. Besides ELF support, some details regarding Mach-O binaries can be extracted as well.\nExample output:\nscanelf -a /bin/ps TYPE PAX PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND FILE ET_EXEC PeMRxS 0755 LE RW- R-- RW- - - LAZY /bin/ps prelink /usr/bin/execstack - display or change if stack is executable /usr/bin/prelink - remaps/relocates calls in ELF files, to speed up the process Example binary file If you want to create a binary yourself, simply create a small C program, and compile it. Here is an example, which opens /tmp/test.txt, reads the contents into a buffer and displays it. Make sure to create the related /tmp/test.txt file.\n#include \u0026lt;stdio.h\u0026gt;; int main(int argc, char **argv) { FILE *fp; char buff[255]; fp = fopen(\u0026#34;/tmp/test.txt\u0026#34;, \u0026#34;r\u0026#34;); fgets(buff, 255, fp); printf(\u0026#34;%s\\n\u0026#34;, buff); fclose(fp); return 0; } This program can be compiled with the gcc command\ngcc -o test test.c\nFrequently Asked Questions What is ABI? ABI is short for Application Binary Interface and specifies a low-level interface between the operating system and a piece of executable code.\nWhat is ELF? ELF is short for Executable and Linkable Format. It is a formal specification that defines how instructions are stored in executable code.\nHow can I see the file type of an unknown file? Use the file command to do the first round of analysis. This command may be able to show the details based on header information or magic data.\nConclusion ELF files are for execution or for linking. Depending on the primary goal, it contains the required segments or sections. Segments are viewed by the kernel and mapped into memory (using mmap). Sections are viewed by the linker to create executable code or shared objects.\nThe ELF file type is very flexible and provides support for multiple CPU types, machine architectures, and operating systems. It is also very extensible: each file is differently constructed, depending on the required parts.\nHeaders form an important part of the file, describing exactly the contents of an ELF file. By using the right tools, you can gain a basic understanding of the purpose of the file. From there on, you can further inspect the binaries. This can be done by determining the related functions it uses or strings stored in the file. A great start for those who are into malware research, or want to know better how processes behave (or not behave!).\nMore resources If you like to know more about ELF and reverse engineering, you might like the work we are doing at Linux Security Expert. Part of a training program, we have a reverse engineering module with practical lab tasks.\nFor those who like reading, a good in-depth document: ELF Format and the ELF document authored by Brian Raiter (ELFkickers).\nTip: If you like to get better in the analyzing files and samples, then start using the popular binary analysis tools that are available.\nWas this article useful to you? Become part of our community and help others by sharing the article with your favorite website or on social media. Any questions or feedback? Let it know\n","permalink":"https://linux-audit.com/elf-binaries-on-linux-understanding-and-analysis/","tags":["binaries","core dump","elf","linux","programming","software"],"title":"The 101 of ELF files on Linux: Understanding and Analysis"},{"categories":["Open source"],"contents":"Getting more out of your project (including more users!) Do you have an open source project, yet you feel that it could more users? You are not alone! Many other open source projects have the same problem. The good news is that with only a few steps, you can new and more active users. Time to learn how promotion can be done without the pushy tricks that marketing and salespeople use.\nA steady growth is good for almost all open source project.\nTraction is everything Traction in software projects is similar to speeding up a car. If you give too much gas, you might end at undesired places. Too little, and it takes too long to reach your destiny. The right amount of traction for your project helps you achieve your goals, provide value to your users, and build a great community.\nThere are actually several reasons why most projects don\u0026rsquo;t get the right amount of traction. Typically these are inadequate promotion, lack of continuous development, or low product quality. In all cases, the result is the same: people won\u0026rsquo;t recommend it to others and the project slowly collects dust.\nLet\u0026rsquo;s introduce myself first I have been actively involved with the development of a few open source projects. Also, I made contributions to other projects with code or with feedback. Two of my own projects are Rootkit Hunter and Lynis. As the author, I feel it is the responsibility of the project owner to properly promote it.\nWho am I? My name is Michael Boelen , owner of security firm CISOfy, blogger, and believer in the value of open source. I\u0026rsquo;m sharing my experiences over the past years, to help you improve the promotion of your project(s). This blog post was created to answer my own Quora question \u0026ldquo;How can I promote an open source software project even more?\u0026rdquo;. Let\u0026rsquo;s continue with the first things you can do, to make your software successful as well.\nPromotion tactics and lessons Being noticed as a person Software is not just about the project itself. One of the biggest lessons to learn is that people are often interested in the person or team behind a project. So the ideal situation is that you get to talk about your project and also to talk about yourself. For this to happen, you have to be noticed as a person and get invited to conferences, or speaking gigs. It allows you to share knowledge and talk about your project at the same time. Don\u0026rsquo;t get invited? Take initiative and respond to CFPs, or Call For Papers.\nThis blog post is actually written while being at on an open source event. This is a great example to show why it works: I\u0026rsquo;m here because of a previous talk about Docker security. I was noticed as a speaker and got asked to speak about it again, at another event. Being noticed helps.\nSo when you get the chance to share your knowledge, use the opportunity and go for it. Maybe you don\u0026rsquo;t feel comfortable enough to do public speaking. In that case, there is always the option to do guest blogging or sharing your project as part of social talk during other meetups.\nTakeaways\nBe active and respond to speaking opportunities Have a blog, or do guest blogging sometimes Show the people who are behind the project Did you notice that I introduced myself, at the beginning of this post? While no article is perfect, I\u0026rsquo;m proud of my work. Don\u0026rsquo;t use a vague alias for your precious work, but show who you are!\nGet recognized for your tooling and give a live demo\nProvide value Over the years, I spoke with many people about open source. One question I had is that what differentiates open source projects, from great open source projects. Repeatedly shared was the notion that software should serve a particular purpose. It needs to provide value or, in other words, it should help the user. It might be something that simplifies work or automates repeating work. Whatever it does, it should solve a pain.\nBy providing value and solving pain, it becomes valuable for your users and they keep using it. When it comes to promoting the open source project, make sure it is clear how it helps the user.\nTakeaways\nBe clear on your website what the tool does and how it helps the user Features are great, describing the benefits is even better Provide clear lists, as people prefer that too big chunks of text Create great documentation One of the painful lessons I\u0026rsquo;ve learned is the requirement of great documentation. It totally makes sense to first create a project, see the amount of interest and then create documentation afterward. However, people expect open source projects to be at least decent, also for documentation.\nLacking good documentation is a warning signal for every new user. Unless the tool is really easy to use, you will need some level of documentation. Divide it into multiple levels of experience, from first use to advanced usage of the tool. This way the users of your software know what to read and avoid wasting time. Typical time wasters are unclear websites with dead links or unclear parameters to use within the software.\nTakeaways\nMake documentation easy to understand, especially for first-time users Split documentation in different levels of usage Have others check your documentation and comment on it Be up-to-date People using open source projects don\u0026rsquo;t like outdated projects. A project that was updated more than 1 year ago, will lose new users. It shows them that it is not maintained. This is also seen on GitHub, with many ghost projects roaming around.\nAlso be careful with adding a date to pages. Avoid adding them if they are not really needed. For example, showing a date on documentation pages might give an adverse result. It may give the impression that a page is outdated. In reality, the instructions are actually still applicable and up-to-date.\nTakeaways\nDetermine a release schedule which you are comfortable with Show people that the software is maintained Be careful with the usage of dates Visibility on developer platforms If you have an open source project, get yourself on services like GitHub and Open Hub. Be creative and get an entry on all well-known developer platforms. Put the website in your signature of forums, but only where it makes sense. One word of caution: don\u0026rsquo;t spam, it will backfire.\nTakeaways\nUse a developer platforms to get your software listed Use a signature on forums Don\u0026rsquo;t spam Make a marketing plan Promotional activities are part of marketing and to some extent of the sales process. You continuously have to do branding and selling of your ideas, even if you are sharing open source software for free. So while it might sound strange at first sight, a marketing plan might actually be of great help.\nYou may start by adding a description of a typical user who would use the software. If there are multiple audiences, describe them. Answer the question of why they would use your software and what benefit they have from using it. Another good addition is the locations where you will promote your solution, like the developer platforms, forums, or other websites.\nTakeaways\nOpen source software is no different than other products and needs promotion Promotion is needed, especially in crowded areas Using a marketing plan helps with the promotion and better value proposition (lesson 2) Software quality Most users of software expect a certain amount of quality from software. Quality is a delicate outcome of how well the software is written, the usability of documentation and how often you can repeat an expected outcome. In other words, it is everything which makes the software rock-solid, usable and understandable.\nOne area which helps with software quality is testing. Although this process might not be that entertaining, you might be surprised about the value. Everyone likes new features, but still many would choose stability over features. So be agile and add new features, but ask yourself how it may impact the quality of your software.\nTakeaways\nDeliver stable software Only add something if it adds value If it lowers quality, don\u0026rsquo;t add it Expectations is key Like quality, people have different expectations. These expectations come in different levels:\nRequired Expected Desired Unexpected A good example is that you might require the software to be properly compiled. This is a minimum to even be able to use the software. If it doesn\u0026rsquo;t, people will stop using it. Next level is expected functionality. An anti-virus tool requires an updating mechanism to keep its malware database up-to-date. Then there are the desired functions, which are simply nice to have. Last but not least, there are the unexpected benefits. The software could be fulfilling a completely different goal than intended. A great example is nmap . It is known as a port scanner, is also able to do much more (like vulnerability scanning).\nTakeaways\nPeople have basic expectations of your software Be clear on your website and documentation on what they might expect Tell them what the software is, and optionally what it isn\u0026rsquo;t. Ease of use Do you like hard-to-use software? Of course not, as it would be wasting time. But often we don\u0026rsquo;t think like the users of our software. This results in the software being difficult to use. My biggest lesson was while teaching students, which never heard of my tools before. Suddenly you see how first-time users interact with your software, to my surprise. Basic routines like extracting files may result in a different outcome.\nTakeaways\nTest your software from the beginning Let other people perform testing and collect feedback Create a first-time user guide, which helps in setting up your software Use corporate resources The most popular open source projects have a great community or have a corporate sponsor behind them. Getting a community of users is hard, takes time and provides no guarantees. People simply come and go, as they please. If you want to get your project to the next level and grow, you need promotion. For\nIf you want to get your project to the next level and grow, you need promotion. For that, it really helps if you have a company supporting you. It may even be your own employer, as long as there are no conflicting interests. The company can help with marketing efforts or share your project with customers.\nAnother great resource that companies can provide is outside your own skills. For example, providing graphics design. You might be able to find a volunteer with great design skills, but then are so many more skills you need to get covered.\nTakeaways\nAny promotion goes quicker with the right support Companies have resources to provide, use them wisely The best open source projects have corporate support (look at the Linux kernel!) Use referrals We are social creatures and like to hear thoughts about products from others. For marketing specialists, referrals are a tool to show \u0026ldquo;social proof\u0026rdquo;. Talking about your product is great, but it is even better when others are talking about it.\nWhat I learned over the years is that the easiest way to get referrals is simply to ask for them. If someone tells you they like your software, ask them if you can get them to share a quote about it.\nThe text of a referral is important, as it should reinforce the benefits of your software. Just one or two lines of text will do. It should explain why it is great, or how it helped that particular user. Even better is to combine it with a clear benefit of the software (e.g. \u0026ldquo;Tool X is so easy to use, that I had a report within 2 minutes. We used it so finally solve that 1-year-old issue, for which no one had the solution!\u0026rdquo;).\nTakeaways\nLet others speak about your project Ask simply for a referral A great moment to ask if when you receive a compliment Provide help output Still many open source projects don\u0026rsquo;t give a clear help on how to use a tool. For example, running a command without a parameter might actually start the tool. It is better to show the user clear instructions, even if they provide no parameters.\nAnother option is to intercept the -h or \u0026ndash;help options. If one provides these, be nice and show the user some guidance. Share the most common parameters, or provide some examples. When they are many parameters, add an additional line which hints the user to use the man page.\nTakeaways\nProvide a guiding hand where needed Instead of telling someone an option is invalid, share options on how to proceed Use directions on where to find more help (website, man page, parameters) Build a community of followers Promotion of your fine piece of art can come in many different forms. From speaking at events to having a Twitter account. Still, there is a much more powerful method to reach your audience: email. Yes, email is old and gets ignored sometimes. However, if someone is opting in to receive e-mails from you, most of them will read it sooner or later. It is a great way to tell them you have a new version available and what the highlights are of this new release.\nMost software projects only have a few fanboys. But it is those people that are really in love with your project and like to help. Sometimes they are not able to do the development themselves, but they can do others things. Great examples are writing about the tool or sharing it with others. Even writing a referral for your website might be a good way to help.\nAnother great group of people is the power-users. They are beyond positive standing with your efforts, and able to contribute even more. They have deep insights into the area, may work for big companies with a lot of resources and have a bigger network than you. These people are valuable and most likely your software is making their life easier. Keep listening to their needs and determine how you can satisfy them, without compromising the goals of your project.\nTakeaways\nUse the skills of fanboys and power-users Listen to your community, but keep your personal values in mind Providing training One area I still have on my list is providing more training. With training, I don\u0026rsquo;t mean a full course. Just offer small pieces of help. It may be an introduction guide, a Youtube movie, or demo. Even while showing tools to many people in the past, you will get new users of the software. Real-life training is a great opportunity to demonstrate and get questions in return.\nAnother area of training is to provide new users with a quick start with accessible content and instruction movies. A great example to create a quick movie is using Asciinema. It shows the user what they have to type in and the related screen output. If you have a graphical user interface, then record a small movie and add that to your website.\nTakeaways\nDo real-life training sessions to learn from questions Record small demo\u0026rsquo;s Know your impact If you reached this lesson, you might be overwhelmed with all the options. Especially if your project has been evolved from just a fun exercise and became a hobby. If you create art, including in the form of digital code, people will form opinions about it. Some even start using the most unstable software components in production environments.\nIt is good to ask users of your software how they discovered the project and where they use it. This way you become more aware of the impact you make with your software and may result in valuable referrals. As a bonus, you might even get introduced to new people. I recently learned that Lynis was used in a European satellite project (Galileo). In other words, Lynis helps to secure devices on earth and maybe already in space, who knows!\nTakeaways\nTry to learn about your users and their use cases Ask people how they discovered the project Use this moment of questioning to predict the future and ask what features they like to see Be careful with packaging Lynis had a lot of releases, resulting in work for package and port maintainers. Fortunately, most of the updates simply mean bumping up the version and create a new release. In some cases, like that of Debian and Ubuntu, a new problem raised: stable releases. These and a few other distributions, freeze software packages and link them to an OS release. That means that now a very outdated version of Lynis is in the Long-time support (LTS) release of Ubuntu.\nSoftware packing is an interesting subject, as it helps to automate software installations. The bad thing on Linux is that there are many different file formats and building processes. Before trying to understand them all, make sure that experienced packagers can help you. If you don\u0026rsquo;t do it correctly it might end up in a task which slowly eats up all your time.\nTakeaways\nGet to know the packaging options Perform some test installations yourself (e.g. run your software on 5 different Linux distributions) Outsource packaging to experienced builders Conclusion There are a lot of options available to promote your open source project. This list above might be overwhelming at first until you discover that most of it is marketing. Although most of us don\u0026rsquo;t have a commercial background, it might help to think a little bit more commercially. Use the question \u0026ldquo;if I had to earn money with this, how would I sell it to my customers?\u0026rdquo;. This sales/marketing mindset might help you to improve your promotion efforts.\nEven if you write the best possible software solution in the world, you will need at least some promotion. In this case more is actually better. At the same time ensure that you build software of good quality, get people involved, and keep growing at a steady comfortable pace.\nMore resources Did you like this post? Then see the resources below for more:\nHow to get traction for (your) open source project (presentation, PDF) Grow your community and increase the value of your tool How do you promote your open source tool? Let it know what you are working on! After all, this is about promotion, right?\n","permalink":"https://linux-audit.com/software/how-to-promote-your-open-source-project/","tags":["development","open source","software development"],"title":"How to promote your open source project"},{"categories":["SSH"],"contents":"SSH or Secure Shell is the popular protocol for doing system administration on Linux systems. It runs on most systems, often with its default configuration. As this service opens up a potential gateway into the system, it is one of the steps to hardening a Linux system. This article covers the SSH security tips to secure the OpenSSH service and increase the defenses of the system.\nOpenSSH security OpenSSH is under development by the security fanatics from the OpenBSD project . Every new piece of functionality is created with care, especially when it comes to security . Although there were some vulnerabilities, OpenSSH is fairly secure by default. There are still some steps left that can be improved. During research for the security auditing tool Lynis, we looked also at the available OpenSSH settings. Besides the tests that are now in Lynis, this article is one of the other results of that research.\nWhat will be covered? We will be covering both the server and client configuration. The configuration syntax and settings are based on OpenSSH 7.x. The examples should be working for most Linux distributions like CentOS, Debian, Ubuntu, and RHEL. You can expect this to be also the case for FreeBSD, OpenBSD, and other systems that use OpenSSH. When in doubt, consult your man page. If you discovered an error or exception, let it know via the comments. Your feedback is welcome.\nAfter reading this article, you will know:\nWhere the client settings and server settings are stored How to see the active and default settings How to test your configuration settings Make an informed decision on how to secure SSH Which tools can help audit SSH and apply best practices SSH basics SSH has two parts: the server daemon (sshd) that runs on a system and the client (ssh) used to connect to the server. Typically administration is done by using an SSH client from a workstation. If you are on Windows, then often you will be using something like Putty .\nWhen it comes to the security of the SSH configuration, it is the server part that is the most interesting. For example, is that the server can decide if normal password based logins are allowed or denied. Even if the client has a preference, it is the server to make the final call. The server configuration file is located at /etc/ssh/sshd_config .\nThe client configuration settings can be found in /etc/ssh/ssh_config (system wide) or ~/.ssh/config (per user). Settings can also be specified during the connection by providing a command-line option.\nBefore we start making changes, let\u0026rsquo;s start with some tips to do it right.\nDeployment tips Do (not) use best practices The web is full of blogs and guides that state they are using so-called best practices. A best practice is an effective and good approach and typically agreed on by the experts and by consensus. Unfortunately, many of the blogs and articles are simple copies from other blogs and without the extensive research. So I strongly suggest to look up some background of the blog and author first.\nIf you see just configuration settings without a good explanation, be careful with applying such changes. Some are outdated or simply not relevant. What is the purpose of setting some value when it is already the default or even removed? So whatever you do, apply critical thinking and don\u0026rsquo;t make assumptions. So use best practices, but always test your changes.\nCheck the status of SSH Is this the first time you will change your SSH configuration? Check the status of the SSH daemon and see if the related service is started on boot. When using a distribution with systemd, make sure the daemon is running and enabled.\nsystemctl status ssh.service\nNote: on some Linux distributions the service is named sshd.service.\nThe output should contain the enabled value.\nLoaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)\nTo see if SSH is running, look at the next line.\nActive: active (running) since Mon 2018-06-04 17:18:33 CEST; 1 months 2 days ago\nUse the SSH configuration test If you make changes to your SSH configuration, it makes sense to restart the service. I strongly recommend to always check your configuration (sshd_config) first. This can be done by using the test mode flag. This additional step ensures the syntax and options are correct before you end up with a nonfunctioning service.\nsshd -t\nThis command should not return any text or errors. Here is an example when something does not look good:\nMake sure to test your settings first.\nMaking changes to a remote system Are you connected to the system with SSH and making changes to its configuration? Instead of restarting, consider sending a reload command to the running SSH daemon. This decreases the chance that you lose your connection and can\u0026rsquo;t reconnect.\nFor systems using systemd, use systemctl to reload the SSH service.\nsystemctl reload ssh.service\nThe alternative is to manually send a SIGHUP to the SSH daemon. Do not to send this to any of the child processes, or you will be disconnected.\nkill -HUP 1234\nAnother option is to temporarily run another SSH process on another port, without becoming a daemon process. Specify the full path and use -D together with the -p for the port number. Then ensure that you can access the temporary connection, especially if you are using a firewall with traffic filtering.\n/usr/sbin/sshd -D -p 2222\nUse CTRL + C to stop the process after you are done.\nDeploy in small steps While it makes sense to do a full deployment of your new SSH configuration to all systems, you might want to be careful. One example is that some older SSH clients can\u0026rsquo;t use the newer key types. So have a look at the oldest Linux distributions that are used to get an idea on compatibility issues.\nShow active SSH connections Before applying changes or restarting the daemon, check for any active SSH connections. This can be done with the ss command, the alternative for netstat.\nss -n -o state established '( dport = :22 or sport = :22 )'\nAny established TCP connection will be displayed. By using both dport and sport, we can confirm what connections are active in both directions.\nSecuring the SSH server configuration Preparations Before we start making changes to our configuration, let\u0026rsquo;s make a backup.\ncp /etc/ssh/sshd_config /root/sshd_config\nAfter that is done, it is good to know that each OpenSSH version has its own defaults. New features may have been added, older settings may have disappeared. To know if a specific setting is set, don\u0026rsquo;t rely on the configuration file. Instead, call the SSH daemon with the extended test mode flag -T to show all details.\nsshd -T\nThe output may look something like this:\nShow active and default settings of the OpenSSH daemon\nNote: configuration settings and values are displayed with lowercase characters.\nSSH security settings Use of X11Forwarding The display server on the client might have a higher exposure to be attacked with X11 traffic forwarded. If forwarding of X11 traffic is not needed, disable it:\nX11Forwarding no\nWhy disabling X11Forwarding matters: the X11 protocol was never built with security in mind. As it opens up channel back to the client, the server could send malicious commands back to the client. To protect clients, disable X11Forwarding when it is not needed.\nDisable rhosts While not common anymore, rhosts was a weak method to authenticate systems. It defines a way to trust another system simply by its IP address. By default, the use of rhosts is already disabled. Make sure to check if it really is.\nIgnoreRhosts yes\nDNS hostname checking By default, the SSH server can check if the client connecting maps back to the same combination of hostname and IP address. Use the option UseDNS to perform this basic check as an additional safeguard.\nUseDNS yes\nNote: this option may not work properly in all situations. It could result in an additional delay, as the daemon is waiting for a timeout during the initial connection. Only use this when you are sure your internal DNS is properly configured.\nDisable empty passwords Accounts should be protected and users should be accountable. For this reason, the usage of empty passwords should not be allowed. This can be disabled with the PermitEmptyPasswords option, which is the default.\nPermitEmptyPasswords no\nIf you see this option enabled, then check which user accounts have no password set.\nMaximum authentication attempts To protect against brute-force attacks on the password of a user, limit the number of attempts. This can be done with the MaxAuthTries setting.\nMaxAuthTries 3\nAlso enable monitoring for authentication failures, which starts at the half the number of maximum attempts. Use these authentication failures together with your SIEM solution, or forward them to your security administrator.\nThe SSH server can be configured to be used together with PAM or pluggable authentication modules. By using a set of rules, part of the authentication stack, the number of failed logins can be used to block a particular user. Another option is to define a period to lock the account when this number of attempts has been reached. This way the server can defend better against brute-force attempts to crack a user account and its password.\nWhen limiting the maximum authentication attempts, be aware that public key authentication (see below) can also eat up your number of attempts. If you want to enforce the SSH client (or SCP) to use password-based authentication, use the related options on the command line.\nssh -o PreferredAuthentications=password -o PubkeyAuthentication=no username@system\nPublic key authentication Instead of using a normal password-based login, a better way is using public key authentication. Keys are considered much safer and less prone to brute-force attacks. Disable PasswordAuthentication to force users using keys.\nPubkeyAuthentication yes\nPasswordAuthentication no\nRefer to the article Using SSH keys instead of passwords, to set up key-based authentication.\nDisable root login It is best practice not to log in as the root user. Use a normal user account to initiate your connection instead, together with sudo. Direct root logins may result in bad accountability of the actions performed by this user account.\nPermitRootLogin no\nNewer versions of OpenSSH also support the value without-password. This value refers to methods like public key authentication. If your installation comes with this value, there is no reason to change it.\nSet SSH protocol If you are running an older system, version 1 of the SSH protocol might still be available. This version has weaknesses and should no longer be used. Since version 7.0 of OpenSSH, protocol 1 is automatically disabled during compile time. If your version is older than that, enforce the protocol version:\nProtocol 2\nUsage of AllowUsers and DenyUsers When not all users should have access to the system, limit the number of people who can actually log in. One way is to create a group (e.g. sshusers) and add people to this group. Next set the AllowGroups option to define that only these users can log in.\nOther possibilities include to only allow a few users with the AllowUsers, or specifically deny users and groups with the DenyUsers, or DenyGroups. Whitelisting access, using the \u0026lsquo;default deny\u0026rsquo; principle, is usually better. So when possible, use the AllowUsers or AllowGroups option.\nGood to know: SSH applies the following order to determine if one can log in: DenyUsers, AllowUsers, DenyGroups, finally AllowGroups.\nUse HashKnownHosts Each time the SSH client connects to a server, it will store a related signature (a key) of the server. This information is stored in a file with the name known_hosts. The known_hosts file itself is available in the .ssh subdirectory of the related user (on the client). In the case the signature of the server changes, SSH will protect the user by notifying about this chance. This option is useful but also has a risk. Previously it was common to store the hostname related to the specific host key. This made it easy for worms and other malicious scripts to use this information and spread to other systems, once they had a single system compromised. To counter this, the HashKnownHosts will hash each host, so it\u0026rsquo;s not readable anymore. While being unreadable for the human eye, it still allows SSH to check for the next time you connect to the same system, as the results in the same hash.\nExample output:\n|1|XV5CFMH8LLIQPq7PxdBhGX7I9PA=|VKNLdODsQlJ/j4cvTZncqs9vgh0= ecdsa-sha2-nistp256 AAAAE2VjZHNhLX_\u0026hellip;._dJ/RzzZLH8Hs0UgroC0=\nRestrict allowable commands OpenSSH allows restricting the commands that a user can run via the command option. This is placed in the authorized_keys file, together with the allowable command and other options.\ncommand=\u0026quot;ps\u0026quot;,no-agent-forwarding,no-port-forwarding,no-x11-forwarding, TYPE_OF_KEY KEY COMMENT\nIn the example above, replace the TYPE_OF_KEY, KEY, and COMMENT fields. The values that are to be used are similar to when using public key authentication.\nAdditional restrictions Configure your firewall Besides adjusting the SSH configuration, consider also limiting access by using traffic filtering. A local firewall like iptables or nftables can be used to restrict access to only allowed systems. Restrict access by only allowing those IP addresses that are trusted.\nUse a jump server Bigger environments typically restrict access by using a jump server or jump host. You may be familiar with them with other names like stepping stone server or bastion host. They are then the only systems within the network that are configured to allow access to other systems. That means if you want to do system administration, you always connect first to the jump server. From there you will be connecting with the target system. A great combination with the previous tip to limit access with firewalling.\nOpenSSH client security settings As there are many SSH clients available, it would be impossible to cover them all in this article. Instead, we will have a look at the OpenSSH client tool.\nClient configuration The OpenSSH client has three ways to be configured. They are processed in order and checked for every available configuration setting. The first match wins.\nOptions provided via the command-line Configuration file in the home directory (~/.ssh/config) Configuration file for all users (/etc/ssh/ssh_config) Let\u0026rsquo;s say there is a setting named A. A is configured system-wide (option 3) with the value of \u0026lsquo;True\u0026rsquo;. User michael has it configured (option 2) being \u0026lsquo;False\u0026rsquo;. In that case, the latter would win. The reason is that it is considered before the system-wide configuration.\nSee the default and active client settings Remember the trick to see the settings for the server (sshd -T)? The client has a similar one, although with a different character.\nssh -G abc\nThe \u0026lsquo;abc\u0026rsquo; in this example is just a random hostname. Ok, it is not really that random. You can use anything you want, including a real hostname. The client can use Host and Match blocks to customize the configuration to a group of systems or an individual system. As the host \u0026lsquo;abc\u0026rsquo; does not exist, that means the default settings will be parsed.\nSSH settings for a single system Let\u0026rsquo;s say we have a system with the name secureserver. Instead of running on port 22, it accepts SSH connections on port 2222. Instead of using -p on the command line each time, we can add a Host block to our configuration file. So if you want to do this for your user, create the config file in your home directory, below the .ssh directory (so /home/username/.ssh/config).\nNext step is creating a block and define the related settings that you want to use.\nHost secureserver Hostname hostname.example.org User mynickname Port 2222 MACs hmac-sha2-512 KexAlgorithms curve25519-sha256@libssh.org Indenting with spaces is not required. I would still advise to do it, so you see which settings belong to what host definition.\nOne question remains: What settings should you use in your client configuration file?\nI suggest applying changes that make your daily work easier. So if you prefer security, set strong defaults. If a particular host is using a different SSH port, creating a Host block and overrule it that way. Regarding KexAlgorithms, use the newer algorithms that are available. This strongly depends on OpenSSH version on the other systems. If you have fairly new OpenSSH versions on the server, then the curve25519 is a good option. It is a high-speed elliptic-curve that is considered secure (at this moment).\nTools for SSH security While it is good to manually harden a system, software and the related configurations can change over time. For that reason, it is helpful to perform a regular security scan.\nLynis This open source security tool is an all-rounder when it comes to testing the security of your Linux system. From the boot loader to your web server, it will check as much as it can. It is free to use and written in shell script. Lynis runs on the system itself, so it can look both in the configuration files and the actually loaded configuration. It includes several tests focused on OpenSSH and its configuration, including security-related settings. Findings or possible improvements are displayed on the screen, so you can directly get into action and start hardening your system.\nDownload the tool via GitHub or from the website . Never used the tool before, then use the Get Started guide.\nssh-audit Although slightly outdated, the ssh-audit tool is a great one to have in your toolbox. Instead of testing on the host itself, it can connect to an SSH server via the network. It performs its testing on the selected target and looks at the responses it receives. Based on these responses it can learn about the system and the SSH server. It even knows about particular vulnerabilities and can warn you about them. Download the tool via GitHub and give it a spin.\nSee the Linux Security Expert category SSH configuration scanners for other alternatives.\nResources Read the man page A good resource for SSH configuration settings is the man page. While this sounds like an easy tip, it is actually useful to know the man page is strong and well-maintained. With all the minor differences between releases, you should never assume what a setting does. Instead, read about the setting and see if it has recent additions. Your configuration of two years ago might already be outdated. Combine this knowledge with the output of sshd -T and you should be able to select the right option for your situation.\nReferences The following references were used to create this article:\nOpenSSH website sshd_config man page SSH tests from Lynis project Did this article help you enhancing your SSH configuration? Great! Become part of the community and share it on your favorite website or on social media. Got questions or suggestions? Let me know in the comments.\n","permalink":"https://linux-audit.com/ssh/audit-and-harden-your-ssh-configuration/","tags":["audit","hardening","login","server-security","ss","ssh","ssh daemon","ssh_config","sshd_config"],"title":"OpenSSH security and hardening"},{"categories":["Kernel","Linux"],"contents":"If you run a Linux server, software patching is a task that will have to be performed on a regular basis. Although most programs can be auto-restarted with a tool like needrestart, there is one exception: the kernel.\nWouldn\u0026rsquo;t it be a nice if we could update the kernel without the mandatory reboot? Here is livepatch, the feature of the Linux kernel that makes it possible. Let\u0026rsquo;s discover how it works and if you can use it on your system.\nMaximize uptime with livepatch What is live kernel patching? Live kernel patching is the process of applying security patches to a running Linux kernel without the need for a system reboot. The implementation for Linux is named livepatch. The process of patching a live kernel is a fairly complex process. It can be compared to an open heart surgery. The patient is the kernel itself, and precision and care are needed to get things right. One wrong move and it is game over.\nOne benefit of live patching is the ability to postpone a reboot until scheduled maintenance can be done. This means that the availability of a system can be maximized. Another benefit is that security updates are not just installed but also active immediately. While the live patching has its own risks, at least the known vulnerabilities can be mitigated.\nRequirements for the patch process To allow live patching to work, several requirements need to be met. First of all, the kernel itself requires to support livepatch. Initial support was added in 4.x, so you need an up-to-date kernel. Secondly, your system needs a client tool to retrieve kernel patches and load them. To allow loading the kernel patches, your system needs to be configured to allow loading kernel modules. The kernel patches are typically created by the Linux distribution. It requires some expertise to know how to redirect instruction sets.\nHow does live kernel patching work? There are three features that enable patching a kernel while it is running:\nKernel probes (Kprobes) Function tracing (Ftrace) Livepatching (livepatch) These features each have their own role and work closely together. As the process of live patching has its risks, each needs to be careful. Responsibility will be passed from one to another until the full patching cycle has finished.\nThe holy trinity: Kprobes, Function Tracer, and Livepatch Let\u0026rsquo;s have a look into the three kernel features that make the process of patching possible.\nKprobes Kprobes or kernel probes is a kernel feature that is used by developers to measure the Linux kernel and perform debugging. Kprobes allows to break into kernel routines and at many code addresses. This called a breakpoint and allow the developer to take an action. Such action could be to run a new set of instructions.\nFtrace The next feature is named Function Tracer or Ftrace. It is a powerful framework to measure several aspects within the kernel like events and interrupts. For example, it can measure the latency of specific functions like writing to disk.\nLivepatch Livepatch is the third component. It is also the latest addition to the kernel. With a custom Ftrace handler, it can redirect routines and jump to a patched set of instructions.\nKernel patch creation Live patching starts with making a patch. This means that a specific kernel function needs to be changed. The creation of the patch can be done with a tool like kpatch-build. The result is a kernel module, that is then distributed. When this module is loaded, it ensures that processes that use a particular system call are using the patched version of it. It is similar to a traffic diversion.\nHistory of live patching implementations Although the livepatch functionality was the last missing link to allow live patching, it took some years of development to reach this point. The first working implementation of kernel patching was Ksplice. This project was part of university research by MIT. Four students created the company Ksplice, Inc. to market this new technology. Ksplice (the company) was acquired by Oracle and sold as a separate service for their own Linux distribution.\nIn 2014, Red Hat created kpatch and released it under the GPLv2 license. In the same year, SUSE announced kGraft. Both technologies are very similar with some minor differences. Red Hat\u0026rsquo;s implementation stops the kernel to apply live patching, while kGraft does lazy patching. Where kGraft requires manual patch creation, kpatch allows both manual and automatic patch creation.\nCommercial offerings As the feature is in high demand, most Linux distributions offer the option only as a paid add-on. Technologies like Ksplice, kpatch, and kGraft, are commercially interesting for the vendors. The typical user of the live patching feature is willing to pay a good amount of money for it. Although there are a few exceptions, most users won\u0026rsquo;t have direct access to this technology. Slowly this might be changing, especially now that livepatch landed in the kernel.\nKernel Live Patching Core The implementation that landed in the Linux kernel source tree is named livepatch. It is the best of both worlds from kpatch and kGraft. It is named the Kernel Live Patch Core and therefore available for everyone. Because this feature is now one of the components of the kernel, no custom patches are needed anymore.\nWhich distributions support live patching currently? At this moment it is not easy to test livepatch, as not all kernels are built with support for it, or have the client tooling to add and apply patches. There are different technologies around, like kpatch, ksplice, kGraft, and livepatch. Here is a quick overview of some of the technologies used and their status.\nArch Linux (livepatch, kpatch-git tool) Debian (unknown, maybe Debian 9?) Gentoo (kpatch or ksplice ) Oracle Linux (ksplice) Red Hat Enterprise Linux 7 (kpatch or ksplice) SUSE (kGraft) Ubuntu 16.04 and higher (livepatch) (Something outdated in this list? Let it know in the comments)\nHow to check if livepatch is supported with your kernel? To check if you have support for livepatch in your kernel, check if the CONFIG_HAVE_LIVEPATCH setting is enabled. There are different ways to check for this support, depending on your Linux distribution.\nArch Linux zcat /proc/config.gz | grep LIVEPATCH\nIf it is enabled it will show you CONFIG_HAVE_LIVEPATCH=y.\nUbuntu For systems running Ubuntu, have a look in your /boot directory.\ncat /boot/config-$(uname -r) | grep LIVEPATCH\nLivepatch status via sysfs If livepatch support in the kernel is enabled, then there is also another way to check it. The livepatch entries and patches can be found in the pseudo file system sysfs. Look for the /sys/kernel/livepatch directory.\nls -ld /sys/kernel/livepatch\nAnother option is to peek in the parent kernel directory to see all kernel-related options.\nIf the livepatch directory is present it means you have kernel support enabled. What is next? A patching client!\nLive patching of the Linux kernel To enable live patching, we need a client to perform this duty. The client has the instructions on how to operate on a specific kernel. As said, this is a delicate job, so it cannot be universally applied. For this reason, it is Linux distribution specific. For this blog post, we will use an Ubuntu 16.04 (LTS) system. The client utility is commercial (provided by Canonical). Fortunately, they allow free users to patch up to three systems to be live patched. Before we can use the software, we need to create a token first.\nInstalling the client Using canonical-livepath (Ubuntu) The first step is to install the livepatch utility named canonical-livepatch with snap.\nsudo snap install canonical-livepatch\nThen enable livepatch:\nsudo canonical-livepatch enable [token]\nYou should get a positive response saying \u0026ldquo;Successfully enabled device. Using machine-token: [token]\u0026rdquo;. If not, have a look at the common issues at the bottom of this post.\nNow that the client tool is installed, it is time to use it. Run it with the status command.\ncanonical-livepatch status\nThe Linux kernel fully patched on Ubuntu\nYou can also use the -verbose option to see more information about any applied patches. For example which CVE was involved.\nHow to know if the kernel is patched properly? So it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available.\n1. Using the livepatch directory The first option is to look in the earlier mentioned directory /sys/kernel/livepatch and see if there are any entries in it.\nSo it says it is doing its job, as it is fully patched. Great, but how do we know? As this is an old kernel, we know there are some patches available. The earlier referenced directory** /sys/kernel/livepatch** has the answer.\nkpatch-livepatch-in-action-on-ubuntu.png\nWe can see there is a patch applied. It has the same kernel version. The last number in the directory name refers to the version number displayed in the canonical-livepatch output.\n2. Using the tainted flag Another way to know that a kernel has been patched is via /proc/sys/kernel/tainted and check if the kernel is \u0026rsquo;tainted\u0026rsquo; (value higher than zero). This tells debugger and other tools that the kernel has been altered or adjusted.\ncat /proc/sys/kernel/tainted\nTo get more information who tainted the kernel, use the dmesg command.\ndmesg -T | grep tainted\nWe can use the same command to see more information about livepatch and related details. Interestingly it shows a failed verification during our testing.\nlivepatch-tainting-kernel-dmesg-output.png\nLivepatch caveats If you have your system well-hardened and disabled loading kernel modules, then livepatch won\u0026rsquo;t work. This is because a kernel module is loaded to apply the patching.\nTo determine if your kernel allows loading modules, have a look at the file /proc/sys/kernel/modules_disabled.\ncat /proc/sys/kernel/modules_disabled\nIf this gives you a \u0026lsquo;1\u0026rsquo;, then kernel modules cannot be loaded and livepatch won\u0026rsquo;t work.\nTroubleshooting livepatch errors Connection to daemon failed (Ubuntu) $ canonical-livepatch\n2016/10/19 17:01:26 Error executing enable.\nConnection to the daemon failed: Get http://127.0.0.1/enable: dial unix /var/snap/canonical-livepatch/15/livepatchd.sock: connect: no such file or directory\nThis issue is most likely caused because your snapd package is outdated. Upgrade it first with sudo apt install snapd.\nCommand not found (Ubuntu) sudo: canonical-livepatch: command not found\nMost likely the binary directory for snaps are not in your PATH variable defined. A workaround is referring to the tool by its full path (/snap/bin/canonical-livepatch).\nInterested in learning more about the kernel and its features? We are working on a project to get them all listed our Linux security features page.\nEnjoyed this article and like to do something in return? Share it with others so more people can read it. Happy patching!\n","permalink":"https://linux-audit.com/livepatch-linux-kernel-updates-without-rebooting/","tags":["linux","oracle","reboot","uptime"],"title":"Livepatch: Linux kernel updates without rebooting"},{"categories":["Auditing","Hardening","Linux"],"contents":"Every Linux system will benefit from more security, especially if it contains sensitive data. With so many resources available on the internet, one might think that securing Linux has become easy. We know it is not.\nLinux system hardening takes a good amount of understanding about how the Linux kernel works. It also requires a good understanding of the operating system principles. In this guide, we will help you to get this understanding and provide you with tips and tools. The final result should be a secure Linux server or desktop system.\nAfter completing this guide, you will know more about:\nWhat system hardening means specifically for Linux What steps can be taken to improve the overall security of your system Why technical audits are a powerful way to keep you secure How to do regular technical audits on Linux systems Let\u0026rsquo;s start with Linux hardening!\nLinux system security: terminology Before we start, let\u0026rsquo;s do a quick introduction to the main subjects. After all, good understanding starts with knowing the key concepts.\nAuditing No system can be secure if it was not tested. One of the testing methods is by performing a security audit. An audit is typically focused on business processes or on the implementation of technical security measures. This last type of audit is also called a technical audit.\nCompliance This luxury word is actually nothing more than how close are you to a particular policy document or technical baseline. Your baseline may state that every system should have a firewall. Part of the compliance check is then to test for the presence of a firewall.\nSystem hardening The process of improving your security defenses is called system hardening. This means the addition of new defenses and improving existing ones. It may even include the removal of components, to keep the system tidy and clean.\nLinux hardening steps So with system hardening, we focus on the presence of security measures for your system. There are many technical aspects to it, but there are a few key principles. Let\u0026rsquo;s have a look at them first.\nMinimizing your resources Every system has a footprint. Similar to a real footprint, it is the size that the system when it comes to risk. The bigger the footprint, the more risk that is involved.\nReduce installed software Typically we can remove things on the server that are no longer needed. For example, some software package might have been installed to do testing. If this package and its software components are no longer used, then it typically makes sense to remove it. A software package that is not installed, can not impact our risk.\nDisable and remove unused accounts Too many companies still have accounts active that should not be there at all. That colleague that left, but the manager forgot to request deletion of the accounts and related permissions. Therefore it makes sense to have technical controls in place to disable accounts. If you have a colleague that leaves the company, have a tool like Ansible disable the account.\nFor technical teams, it might be good to have strict rules on the usage of accounts. For example, is a personal account allowed to run software on a system for more than a single task? Too often, a developer or system administrator starts a process with their own user, instead of a functional account. After the colleague leaves the company, the account is terminated. At some moment the processes started by the account stop working and a business process is disrupted.\nRegular audits and cleanups can reduce these risks. So a strict hygiene when it comes to disable and remove unused accounts may help.\nDefault deny When deploying services, go for a \u0026lsquo;default deny\u0026rsquo; type of access. That means no one gets access, except those that are specifically listed. This can apply to file permissions, firewall rules, and access to data. For every new service, consider if this principle can be applied.\nRemove identification and application versions Too many software components proudly share its name and version. While it looks innocent, it provides attackers with valuable data. It is not that hard to obtain the operating system that is used. When also learning about the used software components, it becomes much easier to see if there are specific attacks available. Hiding software banners and version numbers will also stop most automated attack scripts, as they often go on the hunt for a specific version.\nNginx: Hide the nginx version Adding new security measures Prevention or detection? After reducing the footprint of the system, the next step is to add relevant security measures. Typically you want to select them by category first. This category defines if a measure helps with prevention or focuses on detection. For example, an antivirus scanner typically will do detection. If it has on-access scanning and can save your system from an infection, it also helps with prevention. A firewall denies access to unneeded network ports, so this is prevention. While prevention sounds like the best option of the two, that is not necessarily true. This reason is that not everything can be prevented. So security defenses that focus on detection are needed as well.\nTopics of interest When adding new security measures, there is a lot to chose from. Let\u0026rsquo;s look at some of the available technical measures you can take.\nKernel security The Linux kernel itself is responsible for policing who gets access to what resources. This is a difficult task, as there needs to be an optimal balance between performance, stability, and security. The kernel can be configured in two ways. The first is during compilation, the build process to create the kernel and its modules. The second option is using the sysctl command or its related /proc file system. Learning about the available kernel security features may be a valuable step in securing your Linux system.\nSecuring processes and their capabilities Processes are the workers on the system. They typically have a clear task to fulfill, often with some form of data processing being involved. As processes may have access to sensitive, in this area we can make an educated choice how the Linux kernel handles core dumps. Core dumps are files that represent how a part of the memory looked before an application or process crashed. If you are dealing with a system with a lot of sensitive data, then usually you want to restrict the creation of these files.\nNetwork filtering When hardening a Linux system, one of the first steps is to look at the network traffic that comes in and goes out. If you are using a cloud server, then your neighbor systems might not be as friendly as the ones in your own home network. So it is wise to filter out unwanted network traffic, or better, only allow wanted traffic.\nIngress traffic With network traffic, there are two directions possible: incoming or outgoing. Incoming traffic is that from other systems that want to talk with your system. This is also called ingress filtering, where you want to make sure that the source address (the sender) is valid. Let\u0026rsquo;s say your system has two network interfaces. Interface 1 is connected to your internal network (e.g. 192.168.1.0/24) and interface 2 that is connected to your internet connection. When someone pretends to be on your local network, but the traffic was received on interface 2, then something is wrong. By ingress filtering we deny this type of network traffic, to prevent this so-called spoofing attacks.\nEgress traffic Egress filtering, which applies on outgoing traffic, requires a good understanding of the protocols used on your network. Most systems use the following services:\nDNS traffic for resolving names and IP addresses (port 53, UDP and TCP) Outgoing email (port 25, TCP) Time synchronization (port 123, UDP) HTTP and HTTPS for retrieving updates (port 80/443, TCP, sometimes UDP) Filtering all outgoing traffic can be a good way to prevent malicious traffic, especially when filtering outgoing HTTP/HTTPS traffic. It may prevent an attacker to download their malware from some system on the internet.\nUse the localhost interface Linux systems have a loopback interface named lo. Typically the hostname localhost will resolve to the 127.0.0.1 address linked to it. This interface is often used for network-based services that do not have to be publicly available. For example, a web application and a database engine may use a socket file or use this localhost interface to set up a connection. For that reason, you firewall configuration typically will have to allow all traffic on the 127.0.0.0/8 network. Did you know that you can also use an address like 127.1.2.3 as local address?\nWith the ip command you can show the details of this interface.\n# ip addr show lo 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever The output shows that this system has a \u0026rsquo;lo\u0026rsquo; interface. The inet line shows an IPv4 address of 127.0.0.1. It has an address with CIDR notation /8, meaning that 127.*.*.* are linked to it. It also has an IPv6, which is represented by the inet6 line. Both addresses have a scope of \u0026lsquo;host\u0026rsquo;, meaning this address is only available to the system itself. The IPv4 address has a special \u0026rsquo;lo\u0026rsquo; scope, so this address will be marked as the primary loopback address.\nTip: If your system is still using iptables, have a look at ipset. This extension allows iptables to use lists that can be used to block IP addresses and even full network segments.\nEmail and messaging While most Linux systems are not fulfilling the role of a mail server, it is common to see a Mail Transfer Agent (MTA) like Sendmail, Exim, or Postfix. Typical usage includes the delivery of system-related messages to a central mailbox or system administrator. Sendmail was the most popular choice in the early days of Linux. That has been changed with Postfix being in the lead now. Improve Postfix security by applying measures like encryption, spam filtering, and blocklists.\nApplication security Default configuration Most software has to be configured before it can be used. While a default configuration may work, it is not always the best configuration when it comes to security. For example, the MongoDB database engine did not require authentication. The result is that even unauthorized and anonymous users can see all data stored. Not a good way of protecting your precious data.\nSecurity best practices for applications When you are using well-known applications like Apache, MySQL, and Postfix, then you can be fairly sure that there is detailed documentation. Some even include a specific section on security. This alone can be a valuable resource to learn about security principles and how they apply. So have a look at the documentation of any software component you are actively using, especially those listening on a network port. Hunt down the security section and make an action list of what things to secure.\nFocus areas:\nRead application documentation for security measures Restrict network services when possible Use localhost connections for non-public network services Disable default users Set up authentication Use strong passwords Ongoing security measures Keep the system up Most systems have the goal to deliver value to business processes. One of the main pillars of information security is the availability of a system. A system that is down, can be a risk to the business in multiple ways. So set up monitoring with a tool like Nagios, Prometheus, or Zabbix.\nMake backups Create regular backups to ensure the availability of data. When a system goes down for whatever reason, then you have at least the data to do a recovery. It goes without saying, but a backup is as good as its restore. If you can\u0026rsquo;t restore it, then it is not a backup.\nFocus areas:\nCreate regular backups (and test restores) Implement system monitoring Apply thy software patches New software updates are released on a daily basis. They add new features, resolve bugs and security issues. Most package managers on Linux can show the available updates. Some can even show which updates are security-related.\nWhen possible use automatic updates, especially for those packages that are related to security issues. Don\u0026rsquo;t forget to reboot when the kernel is updated. Otherwise, the system remains vulnerable. If you have a critical machine that needs to keep running, consider using live kernel patching with livepatch.\nWhen using a stable version (e.g. Ubuntu LTS), upgrade to the next version before its official support is ended. Don\u0026rsquo;t wait till the latest moment, but plan ahead and perform those upgrades.\nFocus areas:\nPerform regular software updates Apply security updates automatically Using unattended-upgrades on Debian and Ubuntu Automatic security updates with DNF Perform automated audits Almost every system administrator is overwhelmed with the amount of work and activities. While this puts them under some stress, it will also increase the risk that \u0026ldquo;less important\u0026rdquo; things like installing patches are forgotten. Cleaning up a system after an intrusion, or having to install it (again) is usually a waste of time. Switch from being reactive to a more proactive approach. Implement continuous audits, automate controls and use best practices. To secure a Linux system and keep it secure, focus on the right combination of hardening and auditing. This magic combination will be a powerful tool against evildoers.\nFocus areas:\nPerform system health scans (auditing, vulnerability scanning, performance checking) Implement manual checks (focus on one item each time) Suggested resources Security tools With so many things to do in a day, it is easy to forget about security. Fortunately, there are a lot of open source tools available that can assist. Let\u0026rsquo;s say you have a website and use an SSL certificate. There is a security tool available for most parts of the system and its software. So be creative and find a tool for every aspect that you can think of. Don\u0026rsquo;t know where to start? This top 100 of security tools might give you some inspiration.\nDo you have any other resources that are helpful to other readers? Let it know!\n","permalink":"https://linux-audit.com/how-to-secure-linux-systems-auditing-hardening-and-security/","tags":["audit","email","firewall","hardening","how-to","ipset","linux","linux security","system hardening","system security","time synchronization"],"title":"How to secure a Linux system"},{"categories":["Firewall"],"contents":"The Linux community has a continuous drive to enhance the GNU/Linux kernel. When we look at network traffic filtering, we moved from ipchains to iptables. More recently we saw the introduction of nftables. Next in line is BPFILTER, part of the development work for the Linux 4.18 kernel.\nWhat is BPFILTER? BPFILTER is short for BPF based packet filtering framework. In other words, it is a framework that does packet filtering and is based on BPF. Interestingly, BPF itself is an acronym for Berkeley Packet Filter. So it is clear that packet filtering is an important part of this feature.\nTo understand BPFILTER, we need to understand BPF first. The quick introduction to the technique is that it allows user space tools like tcpdump to filter traffic within the kernel. Let\u0026rsquo;s say that you want to see what traffic is received on port 80 (HTTP). We start the tcpdump tool and give it a port number.\ntcpdump port 80\nNow BPF in its turn will only return those packets that match this specified filter. Because it only needs to pass a limited subset of data, overhead is reduced and high performance is achieved.\nSo how does BPF work? Instead of giving user space tools direct access to a raw network, BPF uses a pseudo-device. This means that it is like a controlled staging area. If allowed, BPF allows a tool such as tcpdump to retrieve data that comes from this staging area.\neBPF: Linux BPF implementation With BPF originating from the BSD platform, it might be not surprising that Linux has a slightly different implementation. It uses eBPF, which stands for extended BPF. Since kernel 3.18 this implementation can be used also for non-networking activities like profiling. This is great to measure to perform debugging on processes. The 3.19 kernel (2015) added support to attach to sockets .\nIt is the Linux 4.x series that added interesting new features when it comes to network traffic filtering. For example, kernel version 4.1 (2015) provides ingress and egress filters. This allows us to influence incoming and outgoing traffic. Kernel 4.15 (2018) allows eBPF hooks for Linux Security Modules (LSM).\nSo in short, eBPF is multi-purpose and has become a powerful toolkit used by the Linux developers. No wonder that others have built great tooling around it, to allow performance measurements and troubleshooting. A good example is the work of Brendan Gregg who works for Netflix. Brendan has contributed a lot to BCC (BPF Compiler Collection) , which is a toolkit to that can retrieve data via eBPF. It helps to answer many questions, like:\nWhich TCP connections are active? What are the latencies of requests to disk? Which MySQL queries are slower than the specified threshold? Which security capabilities are checked? What are the slowest EXT4 calls? Which NFS calls are made? And many more\u0026hellip; So with this introduction into BPF and eBPF on Linux, we can see the potential it has for network traffic filtering. Let\u0026rsquo;s move on and dive into BPFILTER.\nCurrent status of BPFILTER The development is currently at an early stage. Much of the work is done by Alexei Starovoitov, Daniel Borkmann, and David S. Miller. They work on the network layer and maintain eBPF. So no surprise that they are closely involved with work on BPFILTER. Some of the recent code can be found at in the bpfilter branch of Alexei.\nRight now, BPFILTER works as follows: it converts netfilter rules used by iptables into BPF programs. These are little instructions that can be attached to parts in the kernel, like the networking stack. The conversion itself is so-called dynamic translation, also known as just-in-time (JIT) compilation or run-time compilation. This means it happens in user space and is executed when it is needed, instead ahead of time.\nBenefits of BPFILTER Due to the JIT compilation, most conversion work happens in user space. This simplifies the work needed in the kernel and allows for easier management of the code. Other benefits that are to be expected include hardware offloading, easier migrations from existing netfilter rules, and better performance.\n","permalink":"https://linux-audit.com/bpfilter-next-generation-linux-firewall/","tags":["bpf","firewall","netfilter","network","tcpdump"],"title":"BPFILTER: the next-generation Linux firewall"},{"categories":["Business","Career"],"contents":"Years ago it was a challenge to find screenshots of devices running Linux. Nowadays, Linux can power phones, TV\u0026rsquo;s, computer systems, mainframes, and many more devices. With more devices, the demand for Linux knowledge will continue to grow. At the same time, the demand for security is higher than ever. All the media attention and regulations like GDPR, asks for more Linux security specialists. In this post, the goal is to answer the question: How to become a Linux security expert?\nWhat is actually an expert? Personally, I find it hard to call myself an expert on any subject. The more you learn about a subject, the better you realize that there is still much more to learn. So I would consider myself not an expert. However, I do know a few things about Linux, security, and the combination of the two. With this post, let\u0026rsquo;s discover the minimum someone should know to get close to the title of Linux security expert. Before we continue, let me warn you first: the road to be a true expert is long and requires a good amount of persistence. I feel there are no shortcuts possible, as you will have to gain both a technical understanding and gain practical experience. Some of which is hard to gain in your current work, resulting in limited testing in your own lab. Learning can be lonely, but it doesn\u0026rsquo;t have to be. Share your knowledge in the format you prefer and make friends along the way. Are you in? Great! Let\u0026rsquo;s have a look at the required basics of Linux.\nKnow your Linux basics Linux is more than just the kernel. It is typically a synonym for the GNU/Linux kernel, system administrations utilities, and the distribution that packaged it. Whatever Linux distribution is your favorite, all are sufficient to learn about the subject. However, it does help if you have access to source code and documentation. This way you can retrieve more information about some subjects like syscalls.\nSubtle differences might be the key to learning To gain a good understanding of Linux, it is actually a good idea to play with several different Linux distributions. Each of them does things slightly different, from the installation process, up to package management. These differences typically reveal also the discussions that people have about so-called best practices. Should you pull in packages via HTTP and then check their signature, or should the transport be also be encrypted by using HTTPS? I will let you make the choice, so let\u0026rsquo;s go back to the subject of Linux distributions. Get some virtual machines ready! If you want to master Linux, then at least try these: Arch, CentOS, Debian, Gentoo, Ubuntu.\nGo with the flow After you installed a Linux system, it becomes more powerful if you add useful packages to it. The selection of the packages mainly depends on the goal of your system. Your Linux desktop system will most likely have a graphical interface, where your test web server may lacking any X Window package. When we want to become better at Linux, it helps to learn the most common tools. One of the reasons is that some packages can be found on most systems, as they provide tools like ps, ls, passwd, sed, or cat. Surprisingly, many veteran Linux users still don\u0026rsquo;t know all of the default available tools. So have a look at your GNU coreutils package, and see what tools it provides. I would not be surprised if you find a tool in there that you never heard of before. No worries, that is exactly the reason to keep learning.\nIt is a popularity contest Besides the essential tools for system administration, there is typically more software installed. If you like to learn more about common tools, the popularity contest of Debian is a good start. It shows what packages are installed the most. This list gives also a good idea of the basic skills that an expert should be knowing. For example, in the top results you will find file system utilities. A basic knowledge in this area is required to set up the system (partitioning, file system, usage of LVM). It also provides a good gateway to security measures, like file system encryption with LUKS. In other words, focus on popular packages and tools, as this gives some guidance on where to start.\nGetting to know all aspects of a Linux system will take time. Some subjects are easy to learn, yet to master. File permissions is a great example. Setting the basic file permissions and understanding them is not that hard. Most users will be able to answer the question if user Michael can access a particular file. But if you add a umask, file ACLs, and a MAC framework like SELinux to the mix, then it becomes harder to answer. That is actually fine, as we typically can be learning by doing things in our own lab, or by learning from the experience of others.\nForget education, go for skills that matter Those who are less experienced in a particular field, often have the idea that you need to remember everything. My take on this is simple: you don\u0026rsquo;t have to. One of the best skills that you can have in this age of time, is searching skills. There is so much to find on the internet, that typically time and our imagination are our only restrictions. If you can find the right search phrases and apply the right filter and selections, almost everything can be found. The power of searching can often also outperform education. And yes, education is useful and can learn you the deeper understanding of a particular field. Still, you will need to search a lot of your life, so you better improve this skill. Now that we covered some of the Linux basics, let\u0026rsquo;s get into security.\nSpecialization in modern times The field of information security is also never-ending. So many things to learn and a daily stream of news surrounding the subject. I feel that it is actually a blessing that there is so much information available. This restricts us to try and specialize in particular subjects like Linux security. After all, the more generic you are, the easier you are to be replaced, forgotten, or ignored. With a specialization in a niche subject, you may be easier recognized as someone who is passionate about the subject. So the overwhelming amount of information can definitely be an advantage. Let\u0026rsquo;s have a look and see how this applies to Linux security in particular.\nLike Linux, information security has its own foundations. Some are related to concepts, others to the human aspects. Technical parts cannot be ignored, including techniques and tooling. A true Linux security expert definitely will need this in daily activities. One of the biggest challenges might be finding the right sources. Some valuable ones will follow later in this article. Let\u0026rsquo;s focus first on the security skills.\nSecurity skills 101 Information security is a field that relies on integrity, confidentiality, and trust. The people that entered the field because you could earn good money, often discovered that it won\u0026rsquo;t work that easily. The typical security person is a little bit paranoid and does not just do business with everyone. We like to gain confidence in the capabilities of a company and determine the level of trust with the people. Integrity is key, so is confidentiality. You don\u0026rsquo;t simply share details about other customers or people.\nTrust, trust, trust As a founder of a security firm (CISOfy), our best customers are the ones that actually took a while before they made the jump and pay for our service and software. Not that they didn\u0026rsquo;t trust us, they were careful. They tested the waters by using the open source tooling, read something on the blog, then got in contact. This might also apply to the learning curve that comes with information security. One does not simply become good in security overnight. You have to show it. Or like the Americans say: walk the talk.\nSo it is good to remember that confidentiality, integrity, and especially are trust is very important. While looking at the work of others, I see so many that have a bad \u0026lsquo;profile\u0026rsquo;. They are good at what they do. At the same time, they lack in their presentation. With that I mean the overall picture they provide to the rest of the world, like what they tweet, listed on their LinkedIn page, or put up online. If you want to have others trust you, get your story right and use your personal name. No more hiding behind nicknames and pseudonyms.\nCertifications: time-wasters or needed in a field of trust? If you are technical and want to become better in information security, then consider doing some certifications. Most of the certifications won\u0026rsquo;t give you superpowers, but they will provide you with a good foundation. The CISSP certification is a good example and typically a nice addition to your resume. And sure, some people will ridicule the certification or tell you its peanuts. Nevertheless, it gives you a good foundation to build on. Most of the covered principles and areas even have their own specialists. Some know everything about risk management, while others do penetration testing with a focus on physical security. In other words, do the exams that you enjoy and increase your skill set.\nTechnical security skills are in high demand. Interestingly, Linux is often used during security assessments, yet not part of the project scope itself. So a security professional may use Kali Linux , yet only to scan Windows hosts. Still, this professional will really benefit from having good Linux skills, including shell scripting and doing some Python development.\nLinux security topics So if we combine Linux and security, let\u0026rsquo;s have a look at the particular topics that one should know. I discovered them over a period of 10+ year and it looks like most are still applicable and will remain applicable.\nDatabase security Digital forensics and incident response Events and logging File and data security Identification, Authentication, Authorization Kernel security Malware Network traffic filtering Remote administration Software patch management Time and scheduling So these are the topics. Let\u0026rsquo;s discuss them briefly first. After that, I will provide you with some resources to continue your research or training.\nDatabase security Proper database administration is a specialty in itself. The average system administrator will be familiar with the basic concepts of databases. For those aiming to learn Linux security, you have to know at least these concepts as well. Building on that knowledge, you then can zoom into the security aspects. Think of securing the database connection, data encryption, transport encryption, and user management. When setting up some test systems in your lab, go at least for MySQL or MariaDB, MongoDB, PostgreSQL, and SQLite. That typically covers 80-90% of the database engines that you may find in the wild. If you are working for a bigger company, consider diving into Oracle databases.\nDigital forensics and incident response Companies and individuals will be implementing system hardening, yet computers will fall to attackers. From backdooring to crypto mining, systems will always be an interesting target. One of the skills that a Linux security expert could use is that of digital forensics. Learn how file systems and memory can be analyzed to find interesting artifacts. Also, you may want to learn about incident response and dealing with break-ins. It will happen one day, so better be prepared for it.\nEvents and logging All systems deal with events, with everything between the initial boot up to the shutdown of the system. Most of the events are not very interesting, but now and then some are. The storage and analysis do truly matter within this area. It is useful to learn about the basics of syslog, how systemd stores it data, and how to forward useful events to a SIEM solution.\nFile and data security Like databases, sensitive data is typically clustered in a few places on the disk. Learn how to find where data is stored, with tools like lsof. Apply file permissions and take care of the ownership. Where possible, reduce the default permissions by setting a strict umask. Data security also involves looking at the used storage and protocols. Consider encrypting a disk with LUKS, when sensitive data is being stored. When using protocols like SMB and NFS, learn about the specifics of these protocols. They can be configured to reduce who can access what and the related permissions.\nIdentification, Authentication, Authorization Upon connecting to a system, users typically should show some proof before they get access. For Linux systems, it is the Pluggable Authentication Modules (PAM) that play an important role. Although PAM configuration is not an easy subject, it is an important one. Most systems will work fine without adjustments, but if you like to set up multi-factor authentication (MFA), you will have to get to learn how PAM really works.\nKernel security Although Linux is typically more than just the kernel, it is still the kernel that has a huge impact on the security posture of the system. It acts as an agent when accessing hardware. It is the kernel that does traffic filtering, allowing access or denying it, and decide which processes get priority over the others. As part of kernel security, it is good to learn about system calls (syscalls) and what they do. They are often referred to in security modules (e.g. seccomp). Speaking about Linux security modules, or LSM, these are useful additions to fine-tuning permissions. Some are added during the compilation of the kernel, yet most of them can be loaded manually as well. The Linux kernel can also be tuned with the so-called sysctl settings. They can be found in the /proc pseudo-filesystem or by using the sysctl command itself.\nMalware One of the most debated subjects for Linux security is the need for anti-malware solutions. From a basic virus scanner, up to full endpoint solutions, it is hard to answer if a system requires them. This is where your expertise comes in. By following the news and latest threats, it is up to you to decide what is required. With ransomware that previously hijacked MongoDB data, we can safely say that today\u0026rsquo;s opinion can change tomorrow. The Linux security expert recognizes this fact and has familiarity with backdoors, rootkits, worms, and other types of malware.\nNetwork traffic filtering Most systems are connected to the network and benefit from some traffic filtering. This can be done with iptables , nftables, or with BPF. Learn the principles behind firewalls and how they apply to Linux.\nUse a default deny policy Learn how to read a firewall configuration Configure proper logging Remote administration Within this area, extensive knowledge of SSH and sudo is very useful. Both are commonly used, especially within enterprise environments. SSH can be optimized and secured in several areas. Become familiar with the main concepts and set up different types of accounts in your test lab.\nPublic key authentication Avoid root logins Set time-outs for active connections Software patch management One of the areas that deserve a lot of attention is software patching. This is the practice of updating existing packages and reduce the number of known vulnerabilities. A good understanding of the available package managers is key here. Familiarity with apt, DNF, pacman, and yum, will cover already a good number of Linux distributions. Besides manual patching, also learn how to optimize systems by applying automatic updates and using livepatch.\nTime and scheduling If there is one topic overlooked easily, it is time. We take it for granted, yet so many things rely on it. This also applies to Linux systems and security in particular. Time synchronization is required to validate authentication services like Kerberos tickets and one-time passwords (OTP and TOTP). It improves the quality of logs and event data, as we need the time to do our forensics and incident response.\nOther topics to track This list of topics covered mostly those that apply to all systems. With a continuous stream of new developments, we can expect new knowledge areas. Related subjects are virtualization, container technology, quantum computing, unikernels, and small computing devices (Internet-of-Things).\nResources to continue your journey There are many websites and tools available to do something with Linux or security. And although the combination of the two reveals fewer resources, there are still quite some good resources left. I\u0026rsquo;ve collected the most important ones that show up regularly or focus on quality.\nBlogs and articles Qualitative articles are a great way to learn about technical aspects. Unfortunately, most articles on the web are not in-depth. They might instruct one to change the system, but don\u0026rsquo;t properly explain the \u0026lsquo;why\u0026rsquo; behind the change. Here are some resources that are in-depth and are good starters:\nLinux audit (this page, Linux hardening, security, compliance) Blog by Paul Moore , about SELinux Blog by Kees Cook , one of the kernel developers working on Linux security LWN security pages , they contain news and articles Training and courses The SEC506 course of SANS is one of the few courses that are available on Linux security. The materials of SANS are known for their high quality, intensive learning, and being highly technical. The possible downside is that these courses are often limited in time. This is because they are given at a particular location, or to be followed online.\nFor those who rather do lab-based training, you might want to follow developments of the aptly named Linux Security Expert project (disclaimer: I\u0026rsquo;m involved). This website has the goal to provide an extensive training program to learn Linux security by doing practical labs. It does not stop there, as it provides tool collections, security professionals, how-tos, and many other tips and tricks.\nFollow some professionals While there is not one true expert on Linux (ok, maybe Linus is), there are many that specialize in some area of Linux. Then are a few that also care about security, making them great specialists to follow.\nBinni Shah (kernel development, malware, shares many good discoveries at Twitter) Daniel Walsh (SELinux, containers, presenter, works for Red Hat) Hal Pomeranz (forensics, Linux security, instructor at SANS) James Tarala (IT auditing, presenter, instructor at SANS) Jay Beale (tool author, instructor, presenter) Jessie Frazelle (Linux kernel development, container technology, presenter) Kees Cook (kernel security) Michael Boelen (author of this article, also tool developer, presenter) More useful online resources Besides these professionals, there are more useful online resources. One of them is @ToolsWatch on Twitter, which is founded by Nabil Ouchn. He covers security tools and is part of the Black Hat Arsenal. If you want to see tools being demoed by the original authors, this is the place to be. Two other resources to discover tools are @KitPloit , which have a huge following on Twitter. Their reviews are minimal and sometimes the tools are not of high quality.\nConclusion If you reached this part of the article, you took a good amount of time to read. Well done! It is fair to say that becoming a true Linux security expert takes time. Even if you master all aspects listed in this post, there is still so much more to learn. This can be achieved by continuous education, including the reading of in-depth articles. Another valuable resource is by actively participating, including writing a blog post or submitting an article. Not only can it boost your own name, but during the research of writing the article, you will (re)learn things.\nI hope that I have inspired you with this blog post. If so, go ahead and share with other students, your teacher, your colleagues, or any social medium. Do you have additional useful resources for those who want to learn about Linux security? Let it know!\n","permalink":"https://linux-audit.com/how-to-become-linux-security-expert/","tags":["blog","how-to","linux","linux security","penetration testing","system hardening"],"title":"How to become a Linux security expert?"},{"categories":["Linux"],"contents":"The year is closing, so it is time to review Linux security. Like last year, we look at the state of Linux security. A collection of the finest moments. Did we forget something important? Let us know in the comments. This post will remain updated in the upcoming weeks.\nAs this post may appear on HN, Reddit, Slashdot, and other high-traffic sites, this post is heavily cached. Comments may show up with some delay.\nJanuary: MongoDB, Debian encrypted folders, tcpdump, and data loss at Gitlab MongoDB databases under attack worldwide The problem with default passwords is that they don\u0026rsquo;t get changed too often. While this may look innocent during development, it becomes a serious issue if it happens in production. Many Internet-of-Things devices got their fair share of attention last years, now it is time for those who set up MongoDB and left things as-is.\nUnlocking encrypted folders on Debian with just one character Debian bug 852751 revealed a serious issue with Cryptkeeper, the encrypted folder manager. The issue? All encrypted folders using Cryptkeeper can all be unlocked with password \u0026lsquo;p\u0026rsquo;. This issue was caused by using the underlying EncFS component incorrectly.\nCryptkeeper itself seems to be unmaintained for a while. The issue was opened on GitHub, which did not get much response from the project owner. After almost 11 months, it is still open. Open source or not, unmaintained software is a serious risk.\nCVE collection for tcpdump The famous tcpdump tool, known for its command-line network traffic analyzer capabilities, got a serious review. At least 11 CVE numbers have been assigned in 2017. One of the lessons to learn: mature utilities like tcpdump may not be as secure as we like to think.\nGitLab.com service outage We get to learn another lesson in the first month of the year: check your backups. The company GitLab experienced data loss and then learned their backup methods did not work. The related news article from The Register, uses a headline that says enough: \u0026ldquo;GitLab.com melts down after wrong directory deleted, backups fail\u0026rdquo;.\nUnfortunately, the related Google Docs document has been deleted. It showed a list of activities and actions to improve. GitLab is known to be open and they did a good job during the event. So for those who missed it, a copy would have been great.\nFebruary: Linux on the desktop, Android Security issues with Linux on the desktop During FOSDEM (Belgium), Hanno Böck shared some common weaknesses on the Linux desktop. These are related to the typical file extensions and media types you would likely see on a desktop system. Activities like watching images or movies, using icons, or listening to music. Parsers and plugins fail to be strict when dealing with data, resulting in some interesting flaws.\nAndroid won: the most vulnerabilities Android was the operating system that had the most security-related flaws in 2016. As it is based on Linux, it is no surprise that Debian and Ubuntu followed the mobile operating system with the second and third place.\nApril: BrickerBot, LinuxKit Killing IoT devices before they become zombies BrickerBot is a new piece of malware to specifically attack weak-configured devices with network capabilities. If access can be achieved, this malware will do several attempts to destroy the device by removing disk partitions and finally reboot the device. The goal is to \u0026lsquo;brick\u0026rsquo; a device, making it useless. This way it can\u0026rsquo;t join a botnet and become another zombie sending out spam or play in a Distributed Denial of Service attack (DDoS). The author with the nickname \u0026ldquo;The Janitor\u0026rdquo; said in interviews that BrickerBot killed more than 10 million devices. Recently the author announced to stop his activities. One of the reasons is to prevent his identity getting revealed.\nBuilding Linux systems securely Although Docker did not invent the container technology, they are one of the driving forces regarding the adoption of containers. While they are busy getting more traction, they also found time to release a new project named LinuxKit . As the name implies, it is a kit. The idea is to make it possible creating portable, lean, and secure Linux subsystems.\nMay: sudo Attack on sudo Most Linux users will be familiar with sudo, the little tool to temporarily receive elevated privileges. Typically such utilities get reviewed for security flaws, as they are a good target to be exploited. It is interesting to note that still issues are found. With CVE number CVE-2017-1000367 , an issue was discovered in the function get_process_ttyname(), resulting in revealing sensitive information.\nJune: Stack Clash, Cryptomining on Raspberry Pi Clashing the stack Qualys reports a discovered issue named Stack Clash . The stack, part of memory management for processes, is incorrectly handled on Linux and several BSD-based operating systems. The attack can still succeed, even with Linux \u0026ldquo;stack guard\u0026rdquo;, a defensive mechanism, in place.\nCryptomining on the Raspberry Pi The Raspberry Pi is a wonderful multipurpose small device. But as always, patching is crucial. This time the malware dubbed Linux.MulDrop.14 is attacking the little devices. Surprisingly to get it work in bitcoin mining.\nRelated article:\nLinux Malware Mines for Cryptocurrency Using Raspberry Pi Devices July: systemd, CIA 0day is not a valid username for systemd A lively discussion emerged on GitHub about if the username \u0026lsquo;0day\u0026rsquo; is valid or not. Because it starts with a number, systemd decides the username is invalid. That might be annoying, but the issue was created as the actual task was performed with root permissions. The related CVE CVE-2017-1000082 was closed as \u0026rsquo;notabug\u0026rsquo;. Yet another item on the list for the systemd opponents ?\nCIA hacking tool: Aeris Ever heard about the Aeris tool? This CIA hacking tool became known to target portable Linux systems including CentOS, Debian, and others. With its goal to exfiltrate data via secured channels, it can be tricky to detect.\nRelated article:\nAchilles, Aeris, and SeaPea Are 3 CIA Tools for Hacking Mac and POSIX Systems August Know of some relevant news about Linux security in this month? Share it in the comments.\nSeptember: Equifax, Linus, Optionsbleed, IoT, TLS, dnsmasq Equifax suffers a serious breach due to outdated Apache Struts installation. On a daily basis, companies and individuals see their data breached. But what if you never gave your data to a company, yet still one of your most sensitive information, your credit scores, are revealed? Research revealed that an unpatched installation of Apache Struts was used to get in. The Apache Software Foundation expressed their opinion.\nLinus showing some love for security A sudden surprise: Linus wants offensive security specialists to join in the development of the Linux kernel. He expressed this at the Open Source Summit. He wants skilled people to use their knowledge and help to improve the security of the kernel, including the use of fuzzing technology. At least a positive signal from the godfather of Linux.\nOptionsbleed Hanno Böck discovered a vulnerability in the Apache HTTP server software. It is a memory-leaking vulnerability that is similar to the OpenSSL Heartbleed bug (April 2014). For that reason, Böck named it Optionsbleed . The related CVE is CVE-2017-9798 . Optionsbleed affects a relatively limited number of servers.\nIoT devices attacked by Linux.ProxyM Nothing new, but several sources report about weaknesses in the Internet-of-Things (IoT) devices. As these are typically running Linux and have internet connectivity, a single vulnerability is enough to make it part of a botnet. In this particular case, Linux.Proxy infects the related devices. Then they are configured to send spam in small batches. Enough to be annoying, yet low enough to prevent easy tracking an infected device.\nLinux 4.13 released with in-kernel TLS support This new release of the kernel provides TLS support directly. This kTLS functionality will do mostly symmetric encryption, while more advanced functions will be kept external. Additional details can be found at an older post from LWN.\nPoor dnsmasq Now and then, a common library or toolkit is audited by researchers. It happens that a set of vulnerabilities are discovered, often there for years. This month the dnsmasq project had to deal with several vulnerabilities. As this project is used for DHCP and DNS on smaller networks, you can bet it is part of many systems and especially embedded decides like routers. Version 2.78 of dnsmasq was released on the 2nd of October. The related Proof-of-Concept scripts to exploit can be found at the GitHub repository of Google.\nOctober: Ransomware, Linux Security Summit Ransomware is coming to Linux With ransomware being a common issue on systems running Windows, it looks like it won\u0026rsquo;t take long that Linux systems will join. On the Gentoo forums, a case of Linux ransomware was discussed. While the forum thread does not go into much depth on the specifics, we can still learn from it. The first lesson is not to run things as root, especially not a web browser. Nothing new there, but it looks like people still keep doing these things.\nLinux Security Summit Summary James Morris, a Linux kernel developer, wrote an extensive post about last month\u0026rsquo;s Linux Security Summit. A great resource to learn about some projects, like the CII badge program, TPM 2.0, and the ongoing kernel self-protection project.\nRelated links:\nCII badge program Kernel Self-Protection Project November: USB, security people USB \u0026ldquo;mess\u0026rdquo; storage November is a bad month for everything related to Linux and USB. A list of at least 14 CVEs appeared on the oss-security list, with many being able to provide a denial-of-service. These issues were found by Google\u0026rsquo;s syzkaller , a kernel fuzzing tool. Apparently, this is just the tip of the iceberg.\nLinus Torvalds: \u0026lsquo;I don\u0026rsquo;t trust security people to do sane things\u0026rsquo; The creator of Linux is known for its opinions and he isn\u0026rsquo;t shy to share them. A lot of discussions emerged after Linux exploded based on a pull-request earlier by Kees Cook.\nDecember: glibc, VLC audit Memory leak and buffer overflow in glibc The GNU C Library, glibc for short, had a buffer overflow (CVE-2017-1000409) in its dynamic loader (ld.so). At the same time a memory leak (CVE-2017-1000408).\nVLC getting a budget for bug bounties VLC, the popular media player, is getting a budget for bug bounties. This budget of 60.000 euros was made available by the Free and Open Source Software Audit (FOSSA) project. A great way to have people put their eyes on the source code of VLC and get rewarded for serious flaws in the program. Another interesting fact is that this bug bounty program is at the same time a proof of concept (PoC) for FOSSA-2 .\nIn other news Cool tools During the year we found several existing and new tools:\nButtercup (password manager) Decentraleyes (local CDN emulation for increased privacy) Kube-Bench (security benchmark testing for Kubernetes) Privacy Badger (browser privacy plugin) Prowler (AWS CIS Benchmark Tool) Radare2 (binary analysis) testssl.sh (TLS/SSL configuration scanner) Vault (storage of secrets) vFeed (vulnerability database) Vuls (agentless vulnerability scanner) Learned at least one new tool from this list? The Linux Security Expert project has a new database with security tools . See the bottom of this post for more details about this project.\nSome other interesting Twitter handles to follow for your daily dose of tools:\n@HackwithGithub @KitPloit @ToolsWatch Linux malware Linux rootkits Rootkits are one of the techniques to hide an intrusion and keep a persistent foothold in the system. Nothing new here. The number of new rootkits declined with several kernel improvements over the years. Yet sometimes a new rootkit shows up, like the Reptile LKM rootkit .\nRansomware and Linux We had expected that this year ransomware would jump to Linux systems. While it makes sense to target the users with a below-average knowledge of computers, we think there might be a business in hijacking Linux servers. After all, creating ransomware for Linux is a trivial task. Most systems already have the required \u0026ldquo;toolkit\u0026rdquo; available to find files and encrypt them (find, openssl).\nSome examples of Linux ransomware:\nBash ransomware CryptoTrooper Web browsers Linux sandboxing improvements in Firefox 57 Firefox is lately pushing on performance and security. With version 57 they include several improvements for Linux, like using seccomp. Seccomp can be used by developers to specify which system calls may be used.\nLinux security experts Want to learn who is active in the field of Linux security? Here are some experts to follow:\nBinni Shah - Linux security and related topics Diogo Mónica - Docker security Dirk Wetter - TLS, SIEM Hanno Böcki - security research Jessie Frazelle - Linux security nerd at Microsoft (her words) Justin Cormack - Docker security and LinuxKit Hal Pomeranz - Linux security and training Kees Cook - kernel hacker Liz Rice - container technology Michael Boelen - Linux security and auditing Michał Zalewski - security tools Nathan McCauley - Docker security NJ OUCHN - CVE Thomas Grafi - kernel development, networking, containers Someone missing who does activities related to Linux security and is active on Twitter? Let us know in the comments.\nThanks This post has been made possible with the help of our community. This includes the readers of the Linux Audit blog, but also users of Lynis and customers of Lynis Enterprise. They keep us involved in their daily struggles to secure, test, and report on the health of their Linux systems. That brings me to the activities we did ourselves in 2017.\nLynis celebrates its 10 years anniversary Many open source tools are abandoned after a few years of their inception. So it is a great pleasure to see when a tool reaches its 10th birthday and at the same time is still maintained. Lynis, the freely available security scanner for Linux, macOS, and other Unix-based systems is going strong.\nLinux security library and training center This year we started with a new resource named Linux Security Expert (or LSE). The website has the goal to bring tools, authors, and training together in one place. In the first phase of the project, we launched the security tools section, including a top 100 of the best security tools . This list is updated weekly and is based on a scored list of the tools. Healthy projects will find their way to the top, giving authors a reason to keep maintaining their favorite tools.\nThanks for tagging along and wishing you a good holiday season and a safe 2018.\nDid you like this post? Share and spread the knowledge.\nNotes Old links may be deleted over time.\n","permalink":"https://linux-audit.com/linux-security/the-state-of-linux-security/the-state-of-linux-security-2017/","tags":["linux","linux security","tools","web browser"],"title":"The state of Linux security in 2017"},{"categories":["System Administration"],"contents":"A common issue with systems running Ubuntu is that may fill up the /boot partition. You might have discovered it when running apt, which refused to work. That is unfortunate, as you also need apt to resolve the issue. After trying several options, we found a way to resolve this catch 22, with just three steps. Opposed to other solutions, you don\u0026rsquo;t need to move files or do other tricky things on your system. Still a word of caution: any tasks you run on your system is your own responsibility. When possible, make backups, snapshots, etc.\nThe error: Unmet dependencies Typically you will discover if the unmet dependencies error shows up. For some reason, one package depends on the other. This typically happens overnight, especially if you use a tool like unattended-upgrade for automatic patching.\n# apt upgrade Reading package lists... Done Building dependency tree Reading state information... Done You might want to run \u0026#39;apt-get -f install\u0026#39; to correct these. The following packages have unmet dependencies: linux-image-extra-4.4.0-93-generic : Depends: linux-image-4.4.0-93-generic but it is not installed linux-image-generic : Depends: linux-image-4.4.0-93-generic but it is not installed Recommends: thermald but it is not installed E: Unmet dependencies. Try using -f. The first step is to confirm that this issue is really caused by a full /boot partition. This can be done using the df command.\ndf -h /boot\nIf the usage column shows 100%, then it is full. Time to take the next steps.\nStep 1: find and purge kernel packages The first step is to find all kernel packages, except the one we are running now. This is done with the inner dpkg command. The outer (first) dpkg command then purges the output that we received.\ndpkg --purge $(dpkg -l linux-{image,image-extra,headers}-\u0026quot;[0-9]*\u0026quot; | awk '/^ii/{print $2}' | grep -ve \u0026quot;$(uname -r | sed -r 's/-[a-z]+//')\u0026quot;)\nTip: run first the dpkg -l command to see what output is returned.\nWhen the command runs, you may still get some errors like no space left on device. That is fine, as that will resolve after a few packages.\nStep 2: add missing packages Now run the suggested apt-get command. Use --yes if you automatically want to answer yes to the related questions asked.\napt --yes -f install\nStep 3: remove unneeded packages Time to clean up by using the autoremove parameter.\napt --yes autoremove\nNow your system should have free space again on the /boot partition.\nDid this resolve your issue as well? Let it know!\n","permalink":"https://linux-audit.com/troubleshooting-full-boot-partition-ubuntu/","tags":["troubleshooting","ubuntu","unattended-upgrades"],"title":"Troubleshooting a full /boot partition on Ubuntu"},{"categories":["Linux"],"contents":"Myth busting: Linux security As the author of Lynis, I have to run several Linux systems for testing Linux security defenses. And if you do something long enough, some get to see you as a Linux security expert. When that happens, you get asked questions. Surprisingly they are often related to some of the myths. Time to share a few I got asked. If you received this link from me directly, then most likely you asked one :)\nLinux systems are not prone to viruses A firewall is not required on Linux Open source software is more secure than proprietary software Software packages from the official repositories are safe Myths Myth: Linux systems are not prone to viruses This first myth might actually be true. Viruses are very rare for Linux. This is also true for Windows and macOS, as this type of malicious software (malware) is not often seen. In the past, MS-DOS and Windows systems got affected by viruses a lot. From innocent versions that let characters fell down on the screen, to viruses that quickly wiped your whole hard drive.\nIf we look for other types of malware, then worms and ransomware are the most active ones. A worm is a type of malware that has the goal of spreading itself as quickly as possible. Ransomware typically makes use of worm-like capabilities to spread, but with the simple goal to find your valuable data. It then encrypts it and then asks you for a ransom. Both types are a threat to most operating systems, including Linux.\nLike with so many things, there is power and weakness in numbers. With more users on a particular platform, the chance that it is targeted will increase. The number of systems powered by Linux is only increasing. From small devices in the category Internet of Things up to servers that power the most active websites in the world. Linux is everywhere and therefore becomes a target. Or maybe we should say, already is. Look at Android, the Linux-based mobile operating system. Most of the weaknesses are simply Linux security flaws or vulnerabilities in software.\nBashCrypt asking a victim to pay (proof of concept)\nMyth: A firewall is not required on Linux Most Linux distributions have definitely improved the baseline security level over the years. Before many unneeded services were installed and activated by default. This means the number of services listening on a network port has also decreased. This still does not warrant the lack of a firewall.\nThere are actually a few types of firewalls. When we speak about a firewall, it is typically the one that does network traffic filtering. Examples include iptables or nftables. Another type is the application level firewall like OpenSnitch . A tool like this will ask per application what connections are allowed.\nEven if your system is not having much running, it is good to filter incoming and outgoing traffic on your systems. This is especially useful to combat worms and other network-based attacks. After all, your system might be a good network citizen, but your network neighbor might be less friendly. Another benefit from adding a firewall is to understand what services need to run on your system. With that knowledge, it becomes easy to define if any incoming UDP or TCP ports need to be opened at all.\nMyth: Open source software is more secure than proprietary software One of the benefits of open source software (OSS) is the availability of the code. Typically this type of software also comes with some level of \u0026ldquo;free\u0026rdquo;: free as in beer, or free as in speech. Now the general consensus is that when the code of the software is available, more people can look into the code and find bugs and security vulnerabilities. While this is true, it doesn\u0026rsquo;t make the software more secure. For that to happen, the developer needs to be skilled and security savvy. Also, other skilled people actually have to look in the code and be able to find any programming flaws.\nSoftware development is hard, as a developer needs to have a good amount of creativity and logic. You will need the latter to make the software do what you intended. The creativity component is important to find edge cases, like unintended behavior. It also helps with finding more efficient ways to solve a problem. Like so many things, there are usually more paths to achieve the same goal. Sometimes a shortcut might be a good way to achieve more efficiency, sometimes it results in a terrible security weakness in your software.\nMyth: Software packages from the official repositories are safe If you only install software packages via the default software repositories, you might think you are safe.This myth goes hand in hand with that open source software would be more secure than propriety software. While some packages might be officially maintained by the Linux distribution itself, there is still a risk. Such software repository usually contains thousands of packages. The chance that one or more contain security vulnerabilities is high. That a software package is officially maintained, needs to be clarified. Typically it means that the Linux distribution will patch known flaws.\nIf you have the chance to install a package, the official repositories are always preferred. This could be the version distributed by your Linux distribution. Another option is the original software developer or company. If they have an official repository, then that is typically also a good and trusted source. Be careful with adding repositories that are maintained by individuals not related to the project. While their efforts are typically well intended, they might lack the time to keep things up-to-date. Worst case you might even end up with software that is altered. Such alteration could be as bad as added backdoor.\nLinux security tips Now that we discussed some of these myths, let\u0026rsquo;s look at some of the options to improve the security defenses of Linux systems.\nOnly install what you really need Software patch management Implement a firewall Perform regular security scans Only install what you really need Most humans are hoarders, especially when it comes to digitally goods. We collect more and more files and applications. If you want to increase your security, it is time to decrease the number of applications you have installed. So are you testing something and done with it? Remove it. Got some applications installed a while ago and didn\u0026rsquo;t touch them in the last months? Consider removing them as well.\nSoftware patch management We can be short about this one: patch, patch, and patch. Every package that is installed, might contain a software bug. Keep them up-to-date and do it automatically when possible. See the additional resources for some links and tools to do so.\nImplement a firewall Filter out as much network traffic as possible. If your system needs to get a dynamic IP address, then allow DHCP requests to be sent and responses to be received. Allow outgoing network protocols that are required, like DNS for name resolving and NTP for time synchronization. Typically you also want to allow outgoing HTTP and HTTPS connections, to be able to browse the web. Most incoming connections can be safely rejected. Usually you will directly see what traffic is really needed. The rest of the traffic can stay out.\nPerform regular security scans If we only had to give one tip, then this one would be it. With a security scanning tool like Lynis, you can measure possible improvements. Such Linux security scanner is like an umbrella. It contains many different security aspects and defenses that are available on Linux systems. From the earlier mentioned ones, up to the more advanced options available.\nAdditional resources Security advice from Linux distributions All Linux distributions had to learn over time on how to deal with security related issues. Most of them created a security guide, which could be a good way to learn more about security. Some of these mentioned Linux security myths are discussed in more detail. Sometimes even with additional steps on what you can do to improve your security level. So invest a little bit of time and read the security related documentation of your distribution.\nOur guides and tips Over the years we have written about many topics related to Linux security. Here are some suggested reads that help with securing Linux systems.\nLinux generic How much system hardening should you do? Livepatch: Linux kernel updates without rebooting Linux kernel security and how to improve it Optimize SSL/TLS for Maximum Security and Speed Postfix Hardening Guide for Security and Privacy Distributions Automatic updates on Debian and Ubuntu Ubuntu Server Hardening Guide: Quick and Secure Check for a required reboot on Debian and Ubuntu systems Compliance In-depth Linux Guide to Achieve PCI DSS Compliance and Certification GDPR Compliance: Technical Requirements for Linux Systems Did we miss some Linux security myth, or got a good tip? We love to hear!\n","permalink":"https://linux-audit.com/linux-security-myths/","tags":["gdpr","linux","linux security","ransomware","virus"],"title":"Linux security myths"},{"categories":null,"contents":"During the time that we are working on this blog, we came across many useful security tools. As the developers of Lynis, we always like to see what else is available. There is a lot: fuzzing tools, malware scanners, port scanners, vulnerability detection solutions, etc. On this page, we will collect a set of tools that are covered here on the blog.\nFor those who want to become a true Linux security expert, we create a list of Linux security tools . This page will provide information on available security tools. That means their purpose, and also the option to compare them.\nPort scanners A port scanner helps with the detection of services. They are used by pentesters and system administrators. Learn what ports are opened on your systems.\nNmap Vulnerability scanners Almost every system has weaknesses. These so-called vulnerabilities can result in data breaches or even data loss. With the right vulnerability scanner, you can find these weaknesses and repair them. Since there are many, each with their own focus, combine them and use them wisely.\nLynis OpenVAS ","permalink":"https://linux-audit.com/security-tools/","tags":["tools"],"title":"Security Tools"},{"categories":["Email"],"contents":"Postfix is a common software component on servers for receiving or sending email. It has a lot of configuration options available, including those to improve your Postfix security. This Postfix security and privacy guide will help with hardening your Postfix configuration.\nAfter you are finished, your system will have improved defenses against spam, abuse, and leaking sensitive data.\nWhy Postfix hardening? Every service that is connected to the internet is sooner or later to be abused by automated scripts. For example, an incorrectly Postfix might send messages to everyone, instead of just your network systems. This type of configuration is called an open relay. It will get your system ending up on multiple blacklists. If it is just a test system, then you are lucky. If your customers are depending on it, then you have something to explain.\nAnother reason for Postfix hardening is the increasing need for privacy. Most of the legacy protocols, SMTP included, did not have security or privacy high on the priority list. These protocols may share data with other systems without any form of protection. This may result in unauthorized people snooping on data, from your local IT administrator to possibly the CIA or NSA.\nPreparation Time to get technical and get the configuration tested. Many hardening guides and blogs forget an important part of system hardening: the preparation. So let\u0026rsquo;s start with that, before making any changes.\nTest the existing Postfix configuration Your current configuration may have errors without you even knowing. So let\u0026rsquo;s first test for that using the postconf command.\npostconf 1\u0026gt; /dev/null\nThe postconf command can be used to display the Postfix configuration, or make changes. In this case, we redirect all normal output (stdout) to the digital trash bin (/dev/null). If your configuration has any errors or warnings, they will show up. Guess what, one of our systems had actually a warning. This was discovered when we implemented a related test in our own auditing tool Lynis.\nUse the postconf utility to test for configuration issues\nIf you get any output, then it is wise to solve these first and restart your Postfix service to see if the error or warning is gone.\nBackup your Postfix configuration It goes without saying, but too often this step is skipped. If you do system hardening, make a backup first. The first backup is to create a copy of the /etc/postfix directory.\ntar czf /root/postfix-$(date \u0026quot;+%F\u0026quot;).tar.gz /etc/postfix\nFor later troubleshooting or comparing configurations, it is also wise to use postconf to store a copy. This one we can easily use together with the diff command.\npostconf \u0026gt; /root/postconf-$(date \u0026quot;+%F\u0026quot;)\nFind your Postfix version postconf mail_version | awk -F\u0026quot; = \u0026quot; '{print $2}'\nAn alternative is to use your package manager to find the version of the \u0026lsquo;postfix\u0026rsquo; package. For Debian and Ubuntu users, this can be achieved with the dpkg command.\ndpkg -l postfix\nPostfix hardening steps With all the preparations taken, it is time to start with the Postfix hardening steps. Each of the steps will change a particular area within Postfix. Some are to prevent information disclosure, others to enhance stability or increase the privacy of the content being sent.\nBasic hardening Disable VRFY (verify) The VRFY command is short for \u0026lsquo;verify\u0026rsquo;. It can be used to see if an email address is valid on the mail server. While this is great for troubleshooting, it also allows others to make educated guesses if an account exists and deliver possibly spam. The VRFY command is not normally not needed for delivery between two mail servers\npostconf -e disable_vrfy_command=yes\nNote: after changing each item, restart or reload Postfix and monitor Postfix for errors. One way to do this is by keeping a watch on the log file.\nNetwork interfaces (inet_interfaces) The first setting to check is the interfaces Postfix is listening to. This setting is called inet_interfaces and by default configured with all. If you just want to relay messages to other systems, like sending outgoing emails, then there is no needed to listen on all network interfaces. Configure Postfix to listen only on the local interface. This can be achieved by setting inet_interface to loopback-only.\npostconf -e inet_interfaces=loopback-only\nTest your configuration after restarting Postfix. In this case, we can use the output from netstat or ss.\nThe ss utility can show the right details when it comes to local services and their possible network visibility\nImportant notes:\nSome changes need a restart of Postfix A reload is not enough when changing the inet_interfaces setting. If you are configuring a system that relays for other systems, then most likely you want to listen on all network interfaces, or just on localhost and the primary network interface where requests come from. Prevent unwanted email relaying The first rule when putting a mail server up is to avoid being an open relay system. An open relay is a system that accepts email from all systems and forwards them. Spammers use these open relays to send out their messages.\nNetworks Relaying is configured with several parameters. The first one is the mynetworks setting, which typically only includes the network addresses of the local network interface (lo).\nmynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128\nIf you want to extend this list, simply add network segments or individual systems. Specify the related network mask, which is /32 for a single IPv4 address, or /128 for IPv6.\nDue to the spaces in this setting, add quotes when using the postconf command.\npostconf -e mynetworks=\u0026quot;127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128\u0026quot;\nDomains The second layer to define what emails to accept. This is called relaying and one of the options is by looking at the destination domain. The related setting for this is relay_domains, which specifies for which domains to accept email in the first place.\nTest Postfix relaying You can test if your configuration is correctly set up by setting up a connection to the other system. Telnet to the other system, and run the following commands\nhelo yourdomain.com mail from:\u0026lt;your.alias@yourdomain.com\u0026gt; rcpt to:\u0026lt;your.personal.mailbox@gmail-or-hostmail.com\u0026gt; data Type a test line and press CTRL+D twice quit Replace the addresses and see if you can relay a message to an address outside your own domain. For example, you could use a Gmail or Hotmail address as the receiver.\nIf things are properly configured to avoid an open relay, you should be getting a relay access denied message.\nRelay access denied (in reply to RCPT TO command) Incoming email configuration Configuration items starting with smtpd refer to the SMTP daemon. This is the daemon that deals with incoming requests.\nEnable HELO People usually greet each other by saying \u0026lsquo;hello\u0026rsquo; or something similar. Mail servers do this with a HELO command, or EHLO, the extended version. Servers that are not using this are typically not properly configured, or simply sending spam.\npostconf -e smtpd_helo_required=yes\nOutgoing email configuration Typically only a few machines will accept incoming email and all other servers need to send out emails. This might be a system notification from the daily cronjobs. Another typical example is a web application that needs to send emails for account activation or password resets.\nConfigure authenticated relaying with a smarthost The first step is to define which system will accept the email. This system is the so-called smarthost, or relayhost in Postfix terms. Use the postconf command to set these settings from the command line, or edit main.cf directly.\nrelayhost = [hostname.example.org]:587 Use the brackets around the hostname or IP address, to prevent MX lookups. The port number can be changed, depending on the required configuration. Some providers use an alternative SMTP port.\nNext step is to enable SASL authentication.\n# Enable SASL authentication smtp_sasl_auth_enable = yes # Disallow any methods that do allow anonymous authentication smtp_sasl_security_options = noanonymous # Define the sasl_passwd file location smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd Now we will edit the /etc/postfix/sasl/sasl_passwd file.\n[mail.example.org]:587 username:password\nThis file can be parsed by postmap to created an optimized version, which is used as the database for lookups.\npostmap /etc/postfix/sasl/sasl_passwd\nThe last part is configuring encryption. To enable this, we have to configure this separately.\n# Enable STARTTLS encryption smtp_use_tls = yes # Location of CA certificates smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt Now restart Postfix, and send a test email.\necho \u0026quot;test\u0026quot; | mail -s \u0026quot;test\u0026quot; me@example.org\nRelated and useful commands Flush mail queue and retry delivering all emails with the postqueue command:\npostqueue -f\nCryptography, encryption, and privacy Enable TLS logging To see the details from TLS, increase the level of Postfix logging. Set smtp_tls_loglevel (outgoing) or smtpd_tls_loglevel (incoming) to the value one (1).\npostconf -e smtp_tls_loglevel=1\nTesting keys You can easily test your SMTP configuration and related ciphers with OpenSSL. One of the areas to test is the strength of the initial connection handshake. This is typically done with the Diffie Hellman (DH) algorithm, that exchanges the cryptographic keys.\necho | openssl s_client -starttls smtp -connect localhost:25 -cipher \u0026quot;EDH\u0026quot; 2\u0026gt;/dev/null | grep -i -e \u0026quot;Server .* key\u0026quot;\nNote: you need at least version 1.02 of OpenSSL, otherwise not all details are displayed. Use openssl version to double check that you are on a recent version.\nThis command should give you two lines of output. The first line is the temporary key and should be at least 1024 bits when using DH, to prevent the Logjam attack .\nServer Temp Key: DH, 2048 bits\nThe second line is the public key size. This will typically be a 2048 bit key (or higher) on modern systems.\nServer public key is 2048 bit\nDid you find this guide useful? Share it, link to it! If you have any questions, let it know!\n","permalink":"https://linux-audit.com/postfix-hardening-guide-for-security-and-privacy/","tags":["email","encryption","hardening","information disclosure","openssl","privacy","ssl"],"title":"Postfix Hardening Guide for Security and Privacy"},{"categories":["System Administration"],"contents":"Introduction Each Linux system has a bunch of processes running. Most of these processes might be familiar to you if you regularly use a command like ps or top to display them. Processes may look like just an item in a list. They are actually complicated pieces of code that are tamed by a memory manager. To truly understand how your system is running, knowledge of process (or memory) management is of great help. So let\u0026rsquo;s make a jump into the internals of Linux by learning the tools at our disposal.\nWhat is a process? Each system has a particular goal it wants to achieve. Such a goal could be providing a website to anonymous visitors all over the world. To enable that, there should be something listening to the individual website requests, process them, and finally send back the related website page. We call this a process and it consists of machine code. These are individual instructions on what the system should do. These instructions include reading an image from the hard disk, sending data via the network interface, or saving an error message in a log file.\nProcesses come in different forms. The most common type is the commands you may type into a shell program. A shell is a \u0026ldquo;wrapper\u0026rdquo; for your Linux console or virtual terminal screen (when using SSH). Most users use the default Bourne Again Shell (or bash). It allows you to type in text, and it will act upon this input. For example, when you type in a command like ls, it sees this as a known command and executes it.\nThe real magic happens when you run commands. In this event, the shell will decide to run a built-in action or start a program from the hard disk. We call these programs on disk a \u0026lsquo;binary\u0026rsquo;. This binary itself is stored with a specific format, which is typically ELF .\nWhat is a Linux daemon? Some processes have the goal to run for a long time on the system in the background. This could be to fulfill requests like scanning an incoming email or sending back a page of a website. These processes are called daemons. Besides the duration, another big difference is that daemons do not need interaction with the terminal. Typically they won\u0026rsquo;t send any data to it but use log files instead. Daemons are often started directly after the operating system started. Most have a \u0026rsquo;d\u0026rsquo; at the end of the process name, to hint that they are a daemon process.\nThe name daemon comes from an experiment based on Maxwell\u0026rsquo;s demon , that had the job of sorting things in the background.\nGood to remember: A daemon is always a process, but not all processes are a daemon\nWhat about services? Typically the term \u0026lsquo;service\u0026rsquo; was used on Windows systems. With the introduction of systemd, this term is now more applicable for Linux as well. A service is a combination of resources to provide some functionality. For example an SSH service, that consists of running the related daemon and any dependencies like networking.\nRunning processes There is a lot of information to gather and show running processes. Common tools for this job include the ps and top command. Let\u0026rsquo;s start with some basic output that you may get from the first command.\nWhat see in this screenshot is a small listing of processes. Nothing really fancy, right?The process name itself, or actual the command, is at the right (CMD column). We see that the first process is\nThe most common thing we look at, is the process name itself or actual the command. This one is displayed at the right (CMD column). We see that the first process is /sbin/init, which is a common system manager for Linux distributions. The other processes have brackets in their name, which is an indicator that they are kernel threads. This screenshot gives a hint for that, as it includes both the PID and PPID columns. The PID is the process identifier, a unique number for a process on that system at that time. The PPID is the parent process identifier. Both the init process and [kthreadd] have a PPID of zero, which means they don\u0026rsquo;t have a parent process. In other words, these processes stand on their own.\nIf we continue to read the output, we see UID in the process listing, which is the user identifier. Typically the user that started this process, or the owner of the process. The small C column is the CPU usage. On most systems you will see that many processes have a zero, meaning there isn\u0026rsquo;t a lot going on. Later in the list of processes, there is a good example: the unattended-upgrade tool is demanding a good share of CPU cycles (67 percent).\nIf we look at the TTY column, we see that many processes have a question mark. This means they are non-interactive and have no need for a terminal. User \u0026lsquo;michael\u0026rsquo; connected via SSH (third line) and got the bash shell (first line). That process has the \u0026rsquo;tty1\u0026rsquo; terminal, which means it is an interactive process. Finally, the STIME refers to when the process was started. It shows just the time if it was today. Otherwise, a date will be included. The TIME column is how much time the process consumed in CPU usage. Processes that are CPU hungry will have a higher number here.\nNote: Linux users typically use -ef for the ps command, where BSD users are familiar with dashless aux. The output is similar, but minor differences may exist between the operating systems, the version of ps, and the related flags used.\nProcess data The Linux kernel is a complicated machine in itself. It forms the bridge between hardware and software. Its primary goal is to make sure that both sides behave while still process as many requests as possible. A challenging task with hardware interrupts calling continuously for attention, and software known to be sometimes less stable than anticipated.\nTo account for everything running on the system, the kernel needs to track every movement on the system. Especially memory management needs attention. The memory is divided into several zones and then provisioned to the running processes. To prevent misusage, there are guards that monitor the requests for more memory. The goal is to prevent running Out Of Memory (OOM). Otherwise, the OOM killer has to be unleashed from its cage and it starts killing processes to free up memory. Other guards ensure one process can\u0026rsquo;t see the data of other processes, which would be bad for security. Similarly, there are protections that prevent memory segments being incorrectly used, like a data-only area that suddenly runs (malicious) machine code.\nSome of the internal data maintained by the kernel is also useful for the system administrator. The pseudo-filesystem /proc is used for this. A directory is created per PID in this filesystem. In Linux, everything is a file. So each directory consists of a bunch of files. Most of them can be viewed by using the cat command.\nSome examples:\ncmdline - displayed with ps comm - command limits - restrictions like maximum file handles mounts - the visible mounts for the process sched - time scheduler details smaps - shared memory details status - process details, ownership, memory usage While it is interesting to review all files, some smart people created tools to read this data and show it in a more friendly way. Let\u0026rsquo;s have a look at that.\nLinux monitoring, troubleshooting, and in-depth analysis To really learn what your system is doing, you need the right tools for the job. Typically you need a combination of tools to find the cause. For example, a system with a high load could have multiple causes. One of them is high IO, meaning data being written to the disk, or send over the network. So troubleshooting may require taking two or three steps before getting closer to the source.\nLet\u0026rsquo;s learn a bit about processes by answering the most common questions. These are situations you may encounter to run a system and keep it stable.\nHow much memory has the system available? The free command allows you to see the memory and swap statistics. This is useful to determine the totals, used, and free memory.\nfree\nHow can I find all the PIDs for a specific program? The first thing that may come in mind, is using the ps command. Then combine it with the grep command and just show the lines that you want to have.\nps -ef | grep nginx\nIf you like to create an one-liner, or put things in a shell script, use the pgrep command.\npgrep nginx\nIf you want to have it formatted (e.g. with a comma between each PID), use it with the -d and -f flag.\npgrep -d',' -f nginx\nHow much memory does a process use? Linux systems will try to use as much memory as possible for performance reasons. So it is good to know that there are a few values of importance here. The first field is RSS, which stands for Resident Set Size. While there are whole books about memory management, you can see this field as the memory it needs to put the program into memory. Typically just a few megabytes or less. Then there is VSZ, the Virtual Memory Size. This is the memory that the program has access to. This is usually a lot more than the RSS. But it doesn\u0026rsquo;t mean all this memory is really in use.\nTo see specifically the details, we can use the ps command for this task. We can even filter the fields we need and tell it which PID (or PIDs) we are interested in.\nps -o vsz,rss,cmd --pid $(pgrep command)\nMake sure that the command returns something, otherwise the **-**pid argument has no values.\nIf you want to keep monitoring a running process, you can use the top command. By providing a specific PID (or multiple), you can filter out the process you are interested in. Great for monitoring database instances, or your favorite web server daemon.\ntop -c -p $(pgrep -d',' -f nginx)\nWhich process has the most disk activity? Disk usage is one of the most common reasons for a high load. In this case, finding the possible culprit is easy with the pidstat command.\npidstat -d\nTo see an actual disk usage, use the iosnoop command.\nWhich new processes are started? The Linux audit framework could be used to monitor for specific system calls. An easier way is to use the execsnoop command. Depending on your distribution you may be able to install this via the package manager.\nexecsnoop\nAny other tools or one-liners you use during troubleshooting? Let it know!\n","permalink":"https://linux-audit.com/running-processes-and-daemons-on-linux-systems/","tags":["performance","processes","ps","troubleshooting"],"title":"Understanding what runs on your Linux system (and why)"},{"categories":["FreeBSD"],"contents":"FreeBSD is definitely another beast than Linux. In some areas, FreeBSD is really a powerful operating system. Package management is maybe not the first one you may think of. Typically FreeBSD users have two options when it comes to installing packages.\nAuditing FreeBSD with pkg audit Ports collection The ports tree allows the administration to build software they need, with the compilation flags he or she prefers. This makes the software optimized and typically the last versions are available. The downside is in the overhead of compiling software, especially with multiple systems involved.\nUsing binary packages Like Linux systems, the concept of precompiled binaries is available to FreeBSD as well. These binaries are packaged together with configuration files, data files, and documentation. Easy to use, to update, and remove. The downside is less customization possibilities.\nIf you use the package manager \u0026lsquo;pkg\u0026rsquo;, you can use the same utility to perform a security scan.\npkg audit -F\nThe pkg audit command will show details about the vulnerability database vuxml and finally any installed packages that are known to be vulnerable. Where available a CVE number is added, which is a unique identifier for software vulnerabilities, together with a link for more information.\nTo only get the package names use the -quiet (or -q) option. Optionally use the -F if you want to refresh the vulnerability data again.\npkg audit -q\nThis option is great for automated solutions. We use this one in our auditing tool Lynis, to see what packages are vulnerable and count them.\nAre you using other tools on FreeBSD to check for vulnerabilities? Let it know!\n","permalink":"https://linux-audit.com/vulnerabilities/vulnerable-packages-on-freebsd-pkg-audit/","tags":["software vulnerabilities","vulnerabilities"],"title":"Vulnerable packages on FreeBSD: pkg audit"},{"categories":["Troubleshooting"],"contents":"Troubleshooting Lynis This document helps with solving most common issues experienced when running Lynis.\nCommon Lynis errors No hostid and/or hostid2 found Some systems do not have the OpenSSH server package installed. In this case, the hostid2 value may be missing. During the upload it may result in an error.\nError: No hostid and/or hostid2 found. Can not upload report file.\nTo see what Lynis discovered, use the show command.\nlynis show hostids\nIf the hostid2 is missing, we can tell Lynis to use one that is generated and unique for that system.\nopenssl rand 1024 | sha256sum | awk '{ print \u0026quot;hostid2=\u0026quot;$1 }'\nAppend the output of this command (hostid2=xxxxxx) to your custom.prf. Then run the show command again. If the value is properly filled, it should allow you to do the upload.\nWarning: if you deploy Lynis with a configuration management tool, make sure that each system has a unique identifier. Systems may otherwise overwrite data from other systems when uploading it to the central system.\nUnknown protocol, please specify (http, https) in profile (update_server_protocol) This error may show up when using the lynis update release command. This is a deprecated command. Use a new version from the software repository .\n","permalink":"https://linux-audit.com/lynis/troubleshooting-guide-for-lynis/","tags":["lynis"],"title":"Troubleshooting guide for Lynis"},{"categories":["System Administration","Time"],"contents":"Having the right time set on a Linux system is important for data synchronization, forensics, and troubleshooting. Next step is to configure the correct time zone. This article will help you:\nSee the current time configuration Learn how to configure the time zone Time zone information We will have a look on how to check and configure the time zone on Linux systems.\nShow current time zone Most new Linux distributions use systemd now. With the timedatectl command we can quickly see the existing time information, including the time zone.\ntimedatectl\nThe timedatectl output might be something like this.\nNow systemd uses a combination of its own configuration files and the well-known ones. For Linux systems, there are typically two files related to the configuration of time zone information.\n/etc/localtime /etc/timezone The differences between localtime and timezone are their format and type of content. Let\u0026rsquo;s start with localtime.\nThe file /etc/localtime is usually is a symbolic link to another file. With the readlink command we can see the related time zone.\nreadlink /etc/localtime\nThe output on a system in The Netherlands would be looking like this:\nFor systems that use systemd, the localtime file also used.\nThe /etc/timezone file is a plain text file. It usually includes the continent and place, unless a more generic setting is used like UTC.\ncat /etc/timezone\nOur system would show \u0026ldquo;Europe/Amsterdam\u0026rdquo;, which is the same value as the localtime file.\nTime zone configuration There are a few ways to configure the time zone on Linux distributions. Here are some of the options:\nUsing timedatectl (systemd) The first option is for all Linux distributions that use systemd. There should be a timedatectl command available and allows you to make the changes.\ntimedatectl set-timezone Europe/Amsterdam\nIf you have multiple systems in different time zones, then UTC would be the best zone to use. This way all systems use the same information.\ntimedatectl set-timezone UTC\nUsing tzselect Using dpkg-reconfigure (Debian/Ubuntu) Systems that are running Debian, Ubuntu, or similar, can use the dpkg-reconfigure tool. An interactive menu will guide you through the configuration.\ndpkg-reconfigure tzdata\nTime zone and scripts The variable TZ is not exported by default on some Linux distributions. You can easily check this by using the export command.\nIf your Linux distribution has a /etc/profile.d directory, then most likely this is the best place to set the time zone. This way all users get the same value when they log in.\nAfter logging in, each user will now have this time zone as the default value. This is actually only the best solution is all users on the systems are actually in the same time zone. Otherwise, you better have users override it via their local ~/.profile (home directory).\nFound another way to set the configuration on your system? Let it know.\n","permalink":"https://linux-audit.com/configure-the-time-zone-tz-on-linux-systems/","tags":["system administration"],"title":"Configure the time zone (TZ) on Linux systems"},{"categories":["Authentication"],"contents":"Most Linux distributions use pluggable authentication modules (PAM). This modular type of configuration allows system administrators to configure and fine-tune the authentication of users. It also defines the behavior on specific events, like providing an invalid user account or password. PAM can use these events to automatically take an action, like locking an account.\nIntroduction to PAM The configuration of PAM is not that hard, but there are risks involved in the process of making changes. An incorrect change could result in unauthorized users being able to log in, or authorized users to be locked out. So before you deploy any change to your production systems, get familiar with the configuration by using a test system. A virtual machine on your own system could be a great way to experiment safely, allowing you to stay connected via the console.\nTo measure the number of failed logins per user, a module is needed to do the counting. Two popular modules for this are pam_tally and pam_tally2, named after tallying. This is the process of counting things. By putting such a module into your PAM stack, failed logins can be measures and an action can be taken based upon the score. Let\u0026rsquo;s have a look on how to configure this measurement.\nDifferences between pam_tally and pam_tally2 If you never configured a tallying module, we suggest using pam_tally2. This newer version has minor differences with the pam_tally module.\nNo more: no_magic_root, reset, no_reset With pam_tally2 these configuration settings have been dropped. If you are migrating, know that some of the options are no longer available.\nfaillog The faillog command shows the number of failed authentication attempts per user. For pam_tally2 this command does not work, and the pam_tally2 command itself should used. Use -a to see all users, or -u to specify which user you are interested in.\nThe modules pam_tally and pam_tally2 both use a slightly different format. For this reason they use different filenames\n/var/log/faillog /var/log/tallylog With these differences in mind, it is time to do some configuration.\nConfiguration settings For both pam_tally and pam_tally2 there is the deny option. This specifies the number of bad logins before taking action.\ndeny=[number]\nAnother useful setting is unlock_time, a timer before a locked user it automatically unlocked.\nunlock_time=[number]\nThe number of seconds that a particular user can\u0026rsquo;t log in. If this setting is not used, the account will be locked till an administrative user (like root) unlocks the account.\nUsing pam_tally2 Open /etc/pam.d/common-auth:\nAdd the following lines before the configuration block starts, so it is the first configuration item.\nauth required pam_tally2.so deny=2 unlock_time=900 This change will be active at the very first login attempt. So it is important to test if you can still log in yourself (via console, SSH, or otherwise). With this configured, most likely you also want to count failed login attempts via SSH. To tell the SSH daemon to use PAM, a few options should be enabled. We can see the active configuration with the sshd -T command.\nsshd -T | grep -E \u0026quot;(challenge|pam)\u0026quot;\nThis should at least show usepam yes and challengeresponseauthentication yes to have SSH work together with PAM and the tallying module. Next step is restarting ssh\nsystemctl restart ssh.service\nFinally, open up a second terminal screen and use an incorrect password when logging in via SSH. This should activate pam_tally and increase the failure counter. This can be checked with the pam_tally2 command itself.\npam_tally2 -u username\nResetting locked users on Linux Users will be automatically unlocked when using a lock time. If you want to enforce this, or unlock a permanently locked user, use the -r option together with the -u option.\npam_tally2 -r -u username\nAlternatives to pam_tally2 If you don\u0026rsquo;t like to fiddle with PAM settings, one of the others options is to use the fail2ban package. This can use the log files of the system to determine incorrect logins and add a system (temporarily) to a blacklist via iptables.\nTroubles setting up, or got it running on another Linux distribution? Use the comments and we will update this guide.\n","permalink":"https://linux-audit.com/authentication/locking-users-after-failed-login-attempts-with-pam_tally2/","tags":["authentication","pam"],"title":"Locking users after X failed login attempts with pam_tally2"},{"categories":["Compliance"],"contents":"What is GDPR? GDPR or General Data Protection Regulation is a regulation to protect personal data from citizens of the European Union (EU). When speaking about stored data, it includes the handling of data at any given time, from the initial creation of the data, until the final deletion of it. One of the important parts is the right to \u0026lsquo;know\u0026rsquo;. That means that individuals can ask what data is stored about them. Another request they may make is that this data is deleted. You may know this from the \u0026ldquo;right to be forgotten\u0026rdquo; which already applied to Google for some years. The GDPR applies to all companies that store personal data from EU citizens. So even if you are based in the US, a happy shopper from the EU will get you in scope.\nThe challenge with regulations like GDPR are the steps you could take on a technical level. While most of the policy makes sense, the translation to action technical implementations steps is nowhere to be found. We created this blog post to get you started with some best practices for Linux systems.\nFor who is GDPR applicable? If you store personal information about citizens from the European Union, GDPR applies to your organization. Organizations that are located in EU, typically have at least personal information about their personnel. If you provide services or products to individuals, the most likely you will also have EU citizens in your database.\nIn that case, you have to take additional measures to protect sensitive personal data. Here are some examples of parties that usually will have to be compliant with GDPR:\nAssociation with memberships Dating sites E-commerce Forums Hosting companies Marketplaces Sports club Web shops As you can see, both commercial and non-commercial entities will have to comply.\nTechnical requirements for GDPR All data starts with the point of entry (creation) and ends with its deletion. In between, there is the transportation, processing, and storage of it. With this regulation, it is not that easy to mandate specific technical controls. The regulation itself deals with safeguarding personal data. Unfortunately, it does not explain how to do this. For that reason, it is good to use the security pillars: Confidentiality, Integrity, and Availability. By applying technical measures to ensure these three principles, we can get closer to complying with the rationale of the GDPR.\nSecurity scanning As there are so many actions that you can take, the first step is getting to know your systems. Even if you already applied some system hardening in the past, there is still a lot to learn by taking regular measurements. These measurements can be done with Lynis, the open source security scanner that Michael Boelen created in 2007. Despite its age, the project is still maintained and light on its requirements to run. Lynis will measure the security defenses on a system and propose room for improvement. Taking regular measurements (daily!) has several benefits. One is that you will learn about the things that can be improved, even if your network or IT landscape changes. The second benefit is that you can show proof that you do regular testing. If you still have a breach, then at least you can\u0026rsquo;t be blamed for not performing these regular tests.\nAuditing and Events One of the important topics in GDPR is dealing with breaches. Systems are as safe as their weakest link, and most likely there are multiple weak links in each network. So how well you try to protect, one should consider that a breach may happen eventually. To detect a possible breach, logging should be configured in the first place. Most Linux processes have this enabled by default, but tuning might be needed. Important areas include failed login attempts. This includes attempts on the console, via SSH, and also for applications that offer authentication.\nBesides logging the need for proper auditing has been increased over the years. In the event where an investigation is needed, you might want to have full details. For example, on what exactly happened on each system. This can be achieved with the Linux audit framework.\nImplementation tips for Linux (auditing and logging) Implement the Linux Audit framework and monitor for suspicious events Set up remote logging, to ensure log files are available and can\u0026rsquo;t be erased by attackers Use a central management interface to collect logging and apply the first level of automatic filtering Availability and Backups When we think of availability of data, the first thing coming to mind might be high-available (HA) software solutions. While that helps with high service uptime, it does not much to protect data in itself. Backups are from a technical point of view more interesting. It starts with creating the backups (safely) and protect them as good as your original data. Your (next) backup solution might need to have a cryptographic library, to encrypt the data. The backup data should be only readable by those having the unlock key.\nOne aspect of backups is often skipped: the restore. And as we know, your backup is as good as your restore. If you can\u0026rsquo;t restore data, your backup is worthless. You can only know how good your backup is by doing regular restores. Consider this a requirement for your backup solution as well, like having the option to perform automatic restores.\nNetwork filtering and firewalls Data should only flow to places where it really needs to be. Most companies already use network firewalls, yet they don\u0026rsquo;t filter traffic between systems in the same network segment. This is a serious risk, as the intrusion of one single system can result in more systems to be breached.\nThe deployment of iptables on Linux systems can be a simple solution to contain data streams to a bare minimum. Depending on the role of the system, allow the protocols related to the services that should be reachable. On top of that, open up the generic management protocols (port 22 for SSH, the ports for monitoring, etc).\nBest practices for network filtering and firewalls Use \u0026ldquo;default deny\u0026rdquo; Keep the firewall updated Log sensitive data streams Perform regular audits of firewall configurations Mark exceptions properly, with an end date or review date Software patch management Almost every software package on this planet has flaws. Fortunately, most of these so-called bugs do not have a huge impact. A small percentage of bugs result in a security issue that can be misused. These are the ones that we know as software vulnerabilities. Almost any Linux distribution has a way to provide software and patches.\nThe first advice is to have a process in place to test and deploy security patches. Where possible use central solutions that help with deployment and automation. A good example is Red Hat Satellite for RHEL, or Canonical Landscape for Ubuntu systems. If you don\u0026rsquo;t use these, then at least script the deployment of security patches, or leverage a tool like unattended-upgrades.\nBest practices for software patching Using staging for testing software Deploy software on a regular basis Apply security patches as quick as possible with automation General GDPR principles and tips The \u0026ldquo;data as cold coffee\u0026rdquo; principle Most people don\u0026rsquo;t like a pot of old (and cold) coffee. When it comes to data, we tend to be on the cautious side and keep storing it for years. Like you shouldn\u0026rsquo;t heat up cold coffee, you should not keep data too long. Reduce the risks of storing sensitive data where you can. For example, delete data when there is no longer a real need to keep it.\nThe period to keep data differs and should be based on the underlying business purpose. For example, if you need to keep it for regulatory reasons, like accounting data and financials. The period could be multiple years in such case. For the purpose of forensics, some data might be useful for months. For example log files and events coming from auditd. So depending on the data, define a clear point from where on data can be safely deleted. So throw away data when possible.\nKeep also hardware and storage in mind. Hardware and storage can contain old data. Proper decommissioning steps should be applied. One of them is secure wiping of data from removable disks and storage media.\nPasswords, passwords, and passwords We all know that strong passwords are better than \u0026ldquo;Welcome01\u0026rdquo;. Still, most systems and software allow you to choose weak passwords. Use a module like pam_cracklib or pam_pwquality to enforce the usage of strong passwords.\nBesides strong passwords, consider using two-factor authentication. This means that you need two different forms of authentication prove your identity. The first one is the combination of your username and password. The second one could be a token generated on your mobile phone. You can use a project like Google Authenticator PAM . This pluggable authentication module uses the common Google Authenticator app. It can be used together with SSH and other forms of authentication.\nFor Linux systems, it is also a good idea to lock out people after a few failed attempts. This limits the risk of brute-force attacks. These type of attacks try continuously to log in. Another benefit is a lower number of events to deal with. Finally, it could be a good reason to watch for other types of attacks.\nAccountability Don\u0026rsquo;t use functional accounts for system administration. Instead, give each administrator their own account. This helps with accounting and keeps everyone accessing the system a little bit more honest. People tend to be more careful when making changes under their own name.\nSecure protocols only The usage of telnet and other plain-text protocols should be avoided. Use safer alternatives like SSH. Add encryption to those services that support it. One of them includes the protocol SMTP, which is used for sending emails. Even if not all mail servers may use encryption at this moment, the big hosters already turned it on. It helps against snooping and possibly also leaking sensitive information.\nDo you have other tips or questions related to GDPR and technical requirements on Linux systems? Let it know!\n","permalink":"https://linux-audit.com/gdpr-compliance-technical-requirements-for-linux-systems/","tags":["compliance","encryption","gdpr","linux"],"title":"GDPR Compliance: Technical Requirements for Linux Systems"},{"categories":["Authentication","Passwords"],"contents":"Linux and password strength One of the options to improve password security is by setting a minimum length. This prevents users from choosing easy passwords. As part of Linux system hardening, you don\u0026rsquo;t want your passwords to be cracked too quickly by modern password crackers.\nConfiguration Let\u0026rsquo;s have a look at how to configure password security and in particular the length and its strength.\nLogin settings The first area where you can set a password length is in /etc/login.defs . The related setting is PASS_MINLEN and already tells us it is about the minimum length of a password. Modern Linux distributions will no longer use this setting and prefer PAM, or pluggable authentication modules.\nIt started with cracklib (PAM) Maybe the first module for configuring password settings was the cracklib module. Primary focus was on testing passwords, preventing users from choosing too simple passwords. With the configuration options it provides, it allows the administrator to define a password policy. One of these items is the minimum password length. Other settings include the usage of special characters, like the usage of capitals and numbers.\nNext is pwquality (PAM) Based upon the foundation of Cracklib, the pwquality module has similar functionality. It is backwards compatible with pam_cracklib.\nThe related configuration file is /etc/security/pwquality.conf .\nMinimum length is not length When we talk about length, a small note should be made. Both modules related to PAM have a specific meaning when it comes to the minimum length. It is a computed value and includes complexity factors from the password itself. For example if and how many capitals, numbers, or special characters it has. To use the minimum length properly, you also have to configure the other settings.\nConfiguration differences between distributions Each distribution uses their own files when it comes to PAM. Here is an overview of the common locations where you can find the PAM configuration files and specifically the setting related to the minimum password length.\nArch Linux: /etc/pam.d/system-auth with pam_pwquality, or per service. CentOS 7: Using /etc/pam.d/system-auth (symlink) and /etc/pam.d/password-auth (symlink) with pam_pwquality Fedora: /etc/pam.d/system-auth-ac with related lines starting with \u0026ldquo;password\u0026rdquo; Gentoo: /etc/pam.d/system-auth using pam_cracklib RHEL 6: /etc/pam.d/passwd using pam_cracklib RHEL 7: /etc/pam.d/passwd using pam_pwquality Ubuntu: /etc/pam.d/common-password using pam_pwquality with the minlen setting. Your Linux distribution or version missing? Share your feedback.\nIf you only want to enforce password complexity on authentication done via SSH, then use the /etc/pam.d/sshd file. If that doesn\u0026rsquo;t exist on your distribution, search for ssh in the /etc/pam.d directory.\nConfiguring PAM should be done with care\nWhen using the pwquality module, there is a separate configuration file available. This file is /etc/security/pwquality.conf and can also be used for the configuration. We suggest picking this one and document it properly. If you would use both options, you might end up with differences. For example, the passwd tool may then use a different setting than other tools or routines.\nTesting your passwords and their strength After tuning the configuration of your system, you may want to test if the changes were properly implemented. There are a few ways to test for this. The first one is using the passwd command and simply change a password of a test user. Try using simple passwords and slightly longer or complicated variations.\nA better method might be using the pwscore tool, that is part of the pwquality tools package (libpwquality-tools). Start the tool and type in a password to have it tested. Good to know is that this tool uses its configuration from the file /etc/security/pwsecurity.conf . If you set a minlen value via PAM, it might not be picked up.\nSometimes simple but long passwords may score high. That doesn\u0026#39;t make them secure!\nGot other tips for our readers? Let it know!\n","permalink":"https://linux-audit.com/authentication/configure-the-minimum-password-length-on-linux-systems/","tags":["authentication","linux","PAM","password"],"title":"Configure the minimum password length on Linux systems"},{"categories":["Linux"],"contents":"Linux security and its developments In the last 10 years, GNU/Linux achieved something some foreseen as almost impossible: powering both the smallest and biggest devices in the world and everything in between. Only the desktop is not a conquered terrain yet.\nThe last years had a great impact on the world. Both from a real-life perspective, as digitally. Some people found their personal details leaked on the internet, others found their software being backdoored. Let\u0026rsquo;s have a look back on what happened lately and what we can expect regarding Linux security.\nWhy this report?\nWith this article, we want to capture the most important events of the last year. By looking back we might be able to better predict what there is to come in the upcoming years. This article is posted on this blog to provide a flexible shell. Any feedback is welcome in the comments section.\nAbout CISOfy\nThis article is created by the people at CISOfy. We focus on Linux and Unix security and created the open source tool Lynis and its bigger brother Lynis Enterprise. Helping you to perform a security scan on your systems and stay compliant with regulations.\n25 years of Linux This year included the celebration of the Linux project. It was 25 years ago that Linus Torvalds shared his initial creation. One of the lessons we can learn from his first announcement is that security had to find its place. You just needed to spawn 64 processes to perform a denial of service. At that time a reasonable defect, considering the age of the project.\nSecurity highlight: backdoors Backdoor in Linux Mint (February 2016) The popular Linux Mint distribution got a bad surprise . Users who downloaded the distribution on the 20th of February picked up a backdoored release.\nWhat happened? The server of the project was apparently breached via WordPress. The attackers were able to put up a new ISO, with a backdoor in it. If your distribution had the file /var/lib/man.cy, then it was confirmed that this was the bad release.\nLessons learned Stop using MD5. If you still use SHA1, then add also the SHA256 or SHA512 hashes.\nLinux kernel security and self-protection A hot topic is around kernel hardening and the concept of \u0026lsquo;self-protection\u0026rsquo;. The kernel should be able to defend itself to a basic set of attacks. Typically these are buffer overflows and result in unauthorized access to memory segments. Fortunately, some of these protections are now being discussed and the first set of patches have been applied to the official kernel sources.\nOne of these examples is the 4.9 release of Linux. The kernel can now enforce proper memory protections, based on the type of data stored in memory. Code memory is marked executable and read-only, with read-only data being marked read-only and non-executable, and writable data as non-executable.\nAnother recent addition is adding guard pages between stacks. Stacks are used for maintaining a list of activities of a process and determine the next step. The kernel has all these process stacks mapped together, with the risk of one process performing stack exhaustion (similar like a buffer overflow but for stacks). If that succeeds, a process can directly influence another process. With the guard pages, this is protected. It kernel will send back a fault and thwart the attack.\nRelevant links\nThwarting Unknown Bugs: Hardening Features in the Mainline Linux Kernel (29 minutes) Kernel self-protection project Relevant kernel parameters\nCONFIG_DEBUG_RODATA CONFIG_DEBUG_SET_MODULE_RONX CONFIG_CPU_SW_DOMAIN_PAN (ARM) CONFIG_ARM64_PAN (ARM64) CONFIG_X86_SMAP (X86) CONFIG_KASAN_INLINE (for testing) CONFIG_KASAN_OUTLINE (for testing) CONFIG_UBSAN Live patching of the kernel The technology of patching a running kernel is not new. Several technologies were being developed over the years:\nKernelCare kexec kGraft (SUSE) kpatch (Red Hat) Ksplice (Ksplice, now Oracle) With support for kGraft in the kernel sources , distributions can now leverage this functionality. When a new security vulnerability hits the kernel, the distribution can create a related patch. This is then loaded as a kernel module and applies a bypass to the affected function that had the vulnerability. Great care should be put into creating these patches as they will change the running kernel. For this same reason, the kernel will mark itself as tainted to reflect this. It is similar to backdooring the kernel, except for a good cause. If you don\u0026rsquo;t allow loading kernel modules, then this technique won\u0026rsquo;t work obviously.\nCanonical announced in October 2016 the availability of using Livepatch in Ubuntu. This service became available to both customers and free users, although limited up to three systems for the latter.\nThe average lifetime of security bugs Kees Cook, currently working for Google, shared an interesting insight regarding the lifetime of security bugs before they are fixed. This can easily between 3 and 6 years for high and critical issues.\nLinux vulnerabilities Like previous years, this year had a fair number of serious vulnerabilities. With differences in timing between discovery and public disclosure, this list is ordered by CVE number.\nCVE-2015-7547 - glibc Issues in glibc, a very generic library affecting almost all Linux systems, caused some attention early in the year. Discovered by troubleshooting strange issues with SSH , it was discovered the cause was at another location: glibc.\nCVE-2016-1247 - nginx (root privilege escalation) Rotation of log files on systems running nginx on Debian or derivatives could be tricked into escalating privileges .\nCVE-2016-0636 - OpenJDK An issue in some versions of Java 7 and 8 hit in particular desktops, including those running on Linux. With the tendency of security professionals advising to disable Java and Flash, we wouldn\u0026rsquo;t be surprised that issues with this kind of packages will slowly decrease. Oracle bulletin for CVE-2016-0636 .\nCVE-2016-0800 - DROWN attack The DROWN attack was a discovered weakness with SSLv2. Although many web servers are now properly configured, there are still systems around having it enabled. And even your web server is not vulnerable, it can be if SSLv2 is enabled on another system (e.g. mail), while reusing the same key for the SSL certificate.\nCVE-2016-0728 - 0-day Linux root exploit An issue in the keyrings functionality could trigger a leakage of data. Perception Point, who discovered the issue, had a great write-up (Internet Archive) available. Unfortunately, a direct post on their website is no longer available.\nCVE-2016-5696 - Linux kernel vulnerability for 4.6 Luckily without affecting many servers and desktops, it affected Android 4.4 KitKat and later. This vulnerability could be used to hijack TCP sessions.\nCVE-2016-6662 - Critical issue in MySQL and MariaDB This vulnerability could result in root privileges. An extensive write-up explains how it works.\nCVE-2016-4484 - Linux Disk Encryption Bypass This issue is very similar to the GRUB2 authentication bypass discovered in 2015. This time it resulted in a root shell on the machine. Although you still can\u0026rsquo;t access data of the encrypted disks, it should not be there. This issue was limited to systems running Debian or a derivative.\nCVE-2016-5195 - Dirty COW Copy-on-write issues in memory resulting in \u0026ldquo;dirty COW\u0026rdquo;. This time with another great logo and official website .\nLinux Malware A lot of the things that hit the media were related to malicious software. Malware is not new on Linux and may exist since the beginning. Early 2000\u0026rsquo;s we saw rootkits, backdoored binaries, and an arsenal of tools to crash well-known software. We can say that the quality of most software increased. This is especially true when considering the addition of security settings and an ongoing trend to enable them by default. And while the effectiveness of most rootkits diminished, malware on Linux looks to be growing.\nMirai botnet Botnets are a powerful tool for those who want to perform denial of service attacks, send spam email, or simply harvest bitcoins at the cost of others. Linux has a past of botnet clients, varying from simple IRC clients that could execute commands, up to heavily encrypted binaries with different mechanisms to be controlled by the botnet master. Fortinet disassembled the Mirai.B worm and shared it on their blog.\nCore Infrastructure Initiative The Linux Foundation released funds and energy into making Linux more secure. Not just the Linux kernel, but also commonly used software components like OpenSSL, or supporting other open source projects. This work is done under the Core Infrastructure Initiative, or CII.\nWith CII there are four projects which enhance each other and help projects all over the world. One of them is tooling, like offering the right tools. This helps with reproducible builds, something being used with Debian now. Also fuzzing tools, which throw garbage at tools to detect missing input validation or memory issues. Besides tooling there is education, helping projects to connect and find the right resources when it comes to security.\nThen there are those special projects that need a little bit more attention. For example, because they are used by many other projects, or consist of a library. A flaw like we have seen in glibc can have a high impact due to this relationship with other software. These projects are tracked with the Census project and scored on risk.\nThe last interesting project is the Badge program , giving developers an extensive checklist to score your project example .\nInteresting reads Reproducible builds on Debian Conferences Conferences are a great way to share knowledge and insights. Two particular conferences can be highlighted that really focus on security in the area of Linux and open source.\nO\u0026rsquo;Reilly Security Most security conferences focus on the offensive side, think Black Hat and Defcon. Rarely we see conferences focused on just defensive. O\u0026rsquo;Reilly made the bold move to organize two events, one in New York, the other in Amsterdam. The recordings are available if you have a subscription to Safari.\nLinux Security Summit This yearly summit provides a good insight into the status of Linux security. There is so much to tell and to see. So have a look at the playlist .\nOther interesting reads Can\u0026rsquo;t get enough? Here are some topics we might also like:\nLinux Security: A Closer Look at the Latest Linux Threats (Trend Micro) MiKey linux key logger Did you like this report? Share it on your favorite social media channel.\n","permalink":"https://linux-audit.com/linux-security/the-state-of-linux-security/the-state-of-linux-security-2016/","tags":["linux security","oracle"],"title":"The state of Linux security in 2016"},{"categories":["Interviews"],"contents":"Thoughts from a first-time contributor to open source software In this article, we learn from a first-time contributor to open source. His name is Eric Light and lives in New Zealand. We came in contact via the Lynis project and I interviewed him to share his experiences.\nMB: Thanks for taking the time Eric. Can you describe a little bit about yourself?\nI started working with computers when I was eight years old, back when my uncle gave me an Apple 2e. Since then I’ve grown up through all the Windows iterations from Win3.1. When Windows Vista was released, I made the decision to leave the Windows world, and started using Ubuntu full-time. After a while I moved to Linux Mint, then LMDE, and now I use Debian Sid for pretty much everything.\nI started a small IT company in New Zealand in early 2007, and spent eight years providing small business IT support, Excel/VBA software development, and a bit of Python development. I shut down the business in early 2015 and took a role as the IT Manager of a local department store, which was my first real introduction into enterprise-grade IT systems. I was only there for 14 months before the company closed down, but it gave me the foundation to move into my current role as a Network and Security Administrator.\nMB: Do you consider open source software just to be free software, or is there more to it?\nI absolutely believe Open Source is more than just free software. Of course, open hardware platforms such as Arduino show that it’s more than just software. To me, it feels like more of a world-view and philosophy. I’m not a zealot! I accept that some businesses need to keep in-house software under wraps to maintain their competitive advantage. But overall, Open Source fits better with my perspectives on life. The concept of incremental improvements, distributed among a potentially large community, for the benefit of an even larger community… I find that idea quite pleasing.\nMB: Last year you contributed to the Lynis project. How did you find out about the tool?\nI honestly can’t remember how I found it. It seems so long ago… *gazes wistfully into the distance*\nMB: What made you decide to invest your personal time and become active as a contributor?\nActually, the majority of my contribution time was during my working hours. I was running Lynis as part of my previous role, there were a couple gaps that I wanted to address, and it was easier to contribute back to the project itself, instead of keeping my changes local. For example, Lynis at the time didn’t detect ESET Antivirus, so I worked with Michael to identify the ESET processes for HRDN-7230. Once we did that, and I saw how he made the changes, I went on to address some other items that had value to my employer.\nMB: Lynis was the project to which you contributed for the first time. How was that experience?\nAbsolutely brilliant. It was a combination of firsts, actually: my first real work in shell script, my first time really working with Git, and my first real contribution to FOSS. At the beginning, I submitted a suggestion and Michael worked with me to create the patch himself. The next time, he encouraged me to write the tests myself. I was reluctant because I was quite aware that my skills were lacking, but Michael was supportive and worked closely with me while I learned.\nMB: Did you also try to contribute to other projects?\nNothing so far. I did submit a PR to another project for a simple spelling mistake, but it went unmerged for 9 months so I deleted it.\nMB: Most open source projects want more contributors. What do you feel is important to have contributors like yourself to become active and send in improvements?\nThe biggest drivers for me were:\nA project I was interested in A project with an active maintainer A maintainer who was responsive, attentive, and helpful I think I was quite lucky that my first experience as a contributor was with Michael. The Lynis project really ticked all the boxes for me.\nMB: Any tips for people who never contributed to an open source project before, but are open to it?\nStart with a project that you’re interested in. Then, identify a very tiny contribution that you could make - even just fixing a spelling mistake or adding a sentence to some documentation. The response to your contribution will let you know how responsive and engaging your maintainer is. Once you know that you’ve got support and engagement, then you can feel comfortable putting forward bigger contributions.\nThanks for your time and sharing your thoughts.\nDo you have tips or questions regarding contributing to open source projects? Let it know in the comments.\n","permalink":"https://linux-audit.com/first-time-open-source-contributor-eric-light/","tags":["open source","software development"],"title":"First-time open source contributor: Eric Light"},{"categories":["Firewall","System Administration"],"contents":"Many Linux administrators became familiar with iptables and ip6tables. Less familiar are tools like arptables and ebtables. Meet the successor of them all: nftables, a packet filtering framework, with the goal to replace all the previous ones. After reading this guide you will be able to configure your own firewall configuration. Step by step we will show how nftables work. Although no knowledge of iptables is needed, we will share some differences with iptables where applicable. This way we can avoid inefficient rules.\nIntroduction: Netfilter Before we start with this guide info nftables, it is good to know about netfilter. Both iptables and nftables use the netfilter components in the Linux kernel. This explains also the first two letters from this new traffic filtering solution.\nOne of the flaws in iptables is the slightly cryptic way of expressing which information flows are allowed. For that reason, the nftables syntax is shorter and easier to understand. Instead of saying \u0026ldquo;-p tcp\u0026rdquo;, it simplifies it to just \u0026ldquo;tcp\u0026rdquo;. This syntax is also very similar to what tcpdump is using. This is no surprise, as the project was inspired by the elegant syntax of tcpdump. If you know pf from operating systems FreeBSD or OpenBSD, you may also like the new syntax.\nSuggested read: differences between iptables and nftables explained\nPreparations For this beginners guide, we assume that you have a recent Linux kernel. Support for nftables should also be compiled into the kernel, together with the related nftables modules. To be sure, check if you have the nf_tables kernel module available. and if the nft binary is installed.\nKernel Modules To determine if you have the nf_tables kernel module, use the modinfo command.\nmodinfo nf_tables\nThe output will look something like this:\nCheck if nf_tables module is available on your system.\nUse lsmod to show any active nftables kernel module.\nlsmod | grep nf_tables\nThis should at least reveal the nf_tables modules: nf_tables_inet, nf_tables_ip, nf_tables_ip6. On newer versions, this might be nf_tables_ipv4 and nf_tables_ipv6.\nDisable iptables It is possible to mix iptables and nftables. However, this increases complexity and also the chance to introduce errors. So keep it simple and flush out all iptables rules, and make sure it is not loaded.\niptables -F\nDo the same for IPv6:\nip6tables -F\nEnsure that during system reboots the iptables configuration or modules are no longer loaded.\nKernel and client We already have seen the active kernel modules in the sections before. Here is a big difference with iptables. The intelligence of the rule sets has been moved to the client utility nft. This utility parses the rules and compiles them into a language that the kernel module understands. This way the kernel receives an optimized set of instructions. Another benefit is that you can manipulate the rules, instead of having to flush the configuration with every new change.\nConfiguration of nftables Before showing examples, it is good to know some basic rules regarding the configuration syntax.\nThe hash sign (#) is used for comments, similar to your shell. To combine several commands, use the semicolon (;) sign. To split an instruction into several lines, use a backslash () at the end of the line. Then continue with your nft command on the next one.\nSo in short:\n# = comment ; = more commands or parameters to follow \\ = break a rule into multiple lines Tip: when a statement includes a semicolon, you can tell your shell to ignore it, by adding a backslash.\nVariables Repetition is bad. To simplify things, nftables supports variables. Instead of repeating an interface multiple times, you define it at the beginning of your configuration file. After that, you will be using the variable.\nExample: defining interfaces define ext_if = eth0 define int_if = eth1 define all_if = { $ext_if, $int_if } Tables Within the configuration of nftables, a table is at the top of the rule set. It consists of chains, which are containers for rules. Overview: Tables -\u0026gt; Chains -\u0026gt; Rules.\nThe maximum length of a table name is 27 characters. At this moment you can create a table (add), delete it (delete), display it (list) or empty it (flush).\nAddress Families All objects within nftables have a so called namespace, which includes the address family. This address family specifies what kind of hooks will be applied for further analysis of the information stream. For example this can be ip for IPv4 traffic, or ip6 for IPv6 traffic. As nftables is aware of the ongoing usage of IPv6, it simplifies usage for both protocol families. It does so by combining them both within the inet address family.\nFor filtering arp traffic, we previously used arptables. With nftables that kind of network traffic belongs to the arp address family.\nIf you have configured a bridged interface, you may want to use bridge (previously ebtables). Then there is netdev, which is used for ingress filtering, or traffic coming into the system. It allows for early filtering traffic, before it reaches other filters (below layer 3 on OSI model).\nOverview:\narp bridge inet (= ip + ip6) ip ip6 netdev As you can see, the names are as short as possible. You can combine rule sets for IPv4 and IPv6 traffic with the inet address family. Netdev is one of the latest additions and allows filtering before it is\nChains After creating a table, the next step is to create chains. Chains are containers holding rules, and are of a defined type. Chains can be 1 of the two types: base or non-base. Being a base type chain, it has a related hook in the kernel. With a hook, the related chain can \u0026ldquo;see\u0026rdquo; the traffic, otherwise it can\u0026rsquo;t.\nnft add chain ip traffic-filter output { type filter hook output priority 0 \\; policy accept\\; }\nChain types: base, non-base\nHook: input, output\nRules The basic building blocks of rule in nftables consists of the following components:\nexpression(s) operator action The expressions within a rule are evaluated from left to right. When the first expression matches, it continues with the other parts. If the expression does not result in a positive outcome, the next rule in line will be evaluated.\nExamples\nnft add rule Firewall Incoming ip daddr 192.168.0.1-192.168.0.19 drop\nFor example, tcp dport 22 accept\nAdvanced Configuration Sets A set is a collection of data elements. This could be for example filled with IPv4 addresses, or port numbers. Maximum length of a set name is 15 characters. If you exceed this is, an error will follow:\nCould not process rule: Numerical result out of range Anonymous Sets dport { 22, 23, 80, 443 }\nUsage: Directly used in rules\nNamed Sets nft add set inet blocklist blocklist4-perm { type ipv4_addr \\; } nft add element inet blocklist blocklist4-perm { 192.168.1.21, 192.168.1.22 } nft add rule inet blocklist input ip saddr @blocklist4-perm drop Usage: Filled with data, then referenced in a rule\nMappings A map is used to do a mapping. You use one field, to look up the value of another, and act on that.\nExamples needed\nDictionaries Another type is the dictionary (or verdict maps). They use the structure of a set and are a powerful component within nftables, as they can include the verdict.\nnft add rule ip Firewall Forward ip daddr vmap {\\ 192.168.1.1-192.168.1.10 : jump chain-dmz, \\ 192.168.2.1-192.168.2.99 : jump chain-ssn1, \\ 192.168.2.100-192.168.2.199 : jump chain:ssn2, \\ 192.168.3.1-192.168.3.50 : jump chain-desktops \\ } Traffic Hooks Each type of traffic has one or more possible traffic hooks. They can be used to make more specific filters.\nInterfaces iifname = Incoming interface\noifname = Outgoing interface\nProtocols Basic syntax\n\u0026lt;protocol\u0026gt; \u0026lt;dport/sport\u0026gt; \u0026lt;port\u0026gt; \u0026lt;action\u0026gt; Values\nicmp udp ip tcp dport/sport: destination port or source port. For example SSH running on our system, would indicate port 22 as destination for incoming traffic. So in this case: tcp dport 22\nOutgoing traffic to another server, would be outgoing traffic to the SSH daemon on the target, which would be dport as well.\noifname lo accept\nicmp type {echo-reply} drop\nicmp accept\nudp sport bootpc dport bootps accept\nip daddr 127.0.0.1 tcp dport {http, postgresql, ipp} accept\nudp dport dns accept\ntcp dport {dns, http, ntp, https, 9418} accept\nCreating Tables, Chains and Rules Next step is creating tables.\nnft add table inet incoming-traffic\nNote: if you don\u0026rsquo;t specify inet, the ip address family will be used by default\nWithin a table, we then create a chain. A chain is a container of one or more rules and used for the organization of the rule. In other words, it is a rule set.\nNow we have created our table, we add an input chain.\nnft list table inet incoming-traffic table inet incoming-traffic { chain input { } } nft add chain inet incoming-traffic management nft add rule inet incoming-traffic management tcp dport 22 nft add chain inet incoming-traffic web-traffic nft add rule inet incoming-traffic web-traffic tcp dport 80 counter nft add rule inet incoming-traffic web-traffic tcp dport 443 counter Best Practices for nftables Use clear names Like in the world of software development, you have to use self-declaring names for your tables. Some examples use the name \u0026ldquo;filter\u0026rdquo;, which is confusing on what it is doing specifically.\nFrequently Asked Questions How can I see all tables for IPv4 and IPv6? Use the inet address family when using both IPv4 (ip) and IPv6 (ip6).\nnft list tables inet\nWhy do I get an error when trying to show an existing table? You might have forgotten to specify the address family. Use nft list tables first.\nHow can I see the rule numbers within each table and/or chain? nft list table inet filter-traffic -a\nHow can I export my rules and backup them? Use nft export xml or nft export json\nCommon mistakes Like most firewall types, it is easy to make mistakes. We have collected a few common mistakes, so you can avoid them early on.\nLoading rules without flushing When loading rules from a file, flush them first. Or better, make a backup first by exporting them. Then flush the rules and import them.\nDouble firewall rules in nftables\nSplitting IPv4 and IPv6 nftables has a great facility to combine traffic for IPv4 (ip) and IPv6 (ip6), named \u0026ldquo;inet\u0026rdquo;. This way you can enable incoming traffic for your web server on port 80, for both protocol families.\nMaking rules too complicated Like in life, more rules result in more complexity. Keep things as simple as possible. Most likely you are not the only one who has to understand your firewall rules, so building rules needs some attention. Another important step is documenting specific rules which are not obvious.\nForgetting the protocol family When requesting the list of active tables, the result set might seem to be empty. By default, the nft utility will use the ip protocol family. For example, when using the inet family, this will result in no entries listed.\nErrors Invalid table \u0026lt;cmdline\u0026gt;:1:1-12: Error: Could not process rule: Table 'x' does not exist\nIf you try to list a non-existing table, you will receive this error. Show the tables with list tables.\nnft list tables\nOther useful resources Here are some other resources to use\nQuick reference guide https://lists.netfilter.org/pipermail/netfilter-announce/2014/000211.html https://people.netfilter.org/pablo/netdev0.1/slides/nftables-netdev-2015.pdf https://wiki.gentoo.org/wiki/Nftables#Adding_chains https://lwn.net/Articles/657933/ ","permalink":"https://linux-audit.com/networking/nftables/nftables-beginners-guide-to-traffic-filtering/","tags":["arp","arptables","firewall","iptables","ip6tables","ipv6","linux","netfilter","network","nft","nftables"],"title":"Beginners guide to traffic filtering with nftables"},{"categories":["Lynis"],"contents":"Differences between auditd and Lynis Recently I received the question what the difference is between auditd and Lynis. Both focus on auditing, that part is clear. For someone not familiar with both software tools, the technical differences may not directly be obvious. Time to write about that, for everyone that has the same question.\nComparing functionality Let\u0026rsquo;s start with a quick introduction in both tools.\nAudit daemon Auditd is the daemon process in the Linux Audit Framework, written and maintained by Red Hat. It focuses on logging system events (accounting).\nLynis Lynis performs a security audit of the system. You can compare it with health check, or a yearly checkup for your car.\nWhen to use which tool? Both tools share the \u0026ldquo;auditing\u0026rdquo; part, so this is where the confusion might come in. The important difference is the specific goal you want to achieve.\nIf you want to track events (like if your /etc/passwd file was changed, or setting the time), then you want to use auditd. Lynis on the other hand, would check for incorrect file permissions on a file like /etc/passwd. It does not track changes in the file itself.\nConsider Lynis as a the yearly check for your car and auditd the onboard computer of the car which checks that the engine is not too hot while driving.\nConclusion So if you want to track changes, use auditd. If you want to know if things are properly configured, then use Lynis. And you may have guessed it: if you want to get your system properly secured, you want to use both solutions. After all, they have a different goal.\n","permalink":"https://linux-audit.com/how-are-auditd-and-lynis-different/","tags":["auditd","lynis"],"title":"How are auditd and Lynis different?"},{"categories":["Software"],"contents":"Every system needs running processes to fulfill its primary goal. But sometimes things go wrong and a process may crash. Depending on the configuration of the system a core dump is created. In other words, a memory snapshot of the crashed process is stored. The term core actually refers to the old magnetic core memory from older systems. Although this type of memory is no longer being used, we still use this term on Linux systems. Enough for history, let\u0026rsquo;s configure our Linux system to properly handle core dumps.\nLinux and core dumps Most Linux systems have core dumps enabled by default. As always, there is a trade-off to make here. On one hand, we want to gather data for improved stability and troubleshooting. On the other, we want to limit the debug data and avoid leaking sensitive data.\nThe first option is good for machines where unstable programs need to be investigated, like the workstation of a developer. The second option is better suited for production systems storing or processing sensitive data.\nDisable core dumps It makes sense to disable any core dumps on Linux by default for all your systems. This is because the files take up disk space and may contain sensitive data. So if you don\u0026rsquo;t need the core dumps for troubleshooting purposes, disabling them is a safe option.\nOption 1: ulimit via the configuration file To disable core dumps we need to set a \u0026lsquo;ulimit\u0026rsquo; value. This is done via the /etc/security/limits.conf file and defines some shell specific restrictions.\nGood to know is that there are soft and hard limits. A hard limit is something that never can be overridden, while a soft limit might only be applicable for specific users. If we would like to ensure that no process can create a core dump, we can set them both to zero. Although it may look like a boolean (0 = False, 1 = True), it actually indicates the allowed size.\nsoft core 0 hard core 0 The asterisk sign means it applies to all users. The second column states if we want to use a hard or soft limit, followed by the columns stating the setting and the value.\nOption 2: configure ulimit via profile The values for ulimit can also be set via /etc/profile or as a separate configuration file in the /etc/profile.d directory. The latter is preferred when it is available. For example by creating a file named /etc/profile.d/disable-coredumps.sh.\necho \u0026quot;ulimit -c 0 \u0026gt; /dev/null 2\u0026gt;\u0026amp;1\u0026quot; \u0026gt; /etc/profile.d/disable-coredumps.sh\nThis command adds the setting to a new file and sets both the soft and hard limit to zero. Each user gets this value when logging in.\nBesides ulimit settings, there are also kernel settings to consider. So choosing one of the options is the first step.\nOption 3: disable via systemd When using systemd and the systemd-coredump service, change the coredump.conf file. This file is most likely located at /usr/lib/sysctl.d/50-coredump.conf. As systemd has a set of files, ensure to check the others like:\n/etc/systemd/coredump.conf /etc/systemd/coredump.conf.d/*.conf /run/systemd/coredump.conf.d/*.conf /usr/lib/systemd/coredump.conf.d/*.conf Set the Storage setting to \u0026rsquo;none\u0026rsquo;. Then configure ProcessSizeMax to limited the maximum size to zero.\n[Coredump]\nStorage=none\nProcessSizeMax=0\nTypically it is sufficient to just reload the systemd configuration.\nsystemctl daemon-reload\nIf this still creates a core dump, then reboot the system.\nDisable setuid processes dumping their memory Processes with elevated permissions (or the setuid bit), might be still able to perform a core dump, depending on your other settings. As these processes usually have more access, they might contain more sensitive data segments in memory. So time to change this as well. The behavior can be altered with a sysctl key, or directly via the /proc file system. For permanent settings, the sysctl command and configuration is typically used. A setting is called a \u0026lsquo;key\u0026rsquo;, which has a related value attached to it (also known as a key-value pair).\nTo disable program with the setuid bit to dump, set the fs.suid_dumpable to zero.\necho \u0026quot;fs.suid_dumpable=0\u0026quot; \u0026gt;\u0026gt; /etc/sysctl.conf\nReload the sysctl configuration with the -p flag to activate any changes you made.\nsysctl -p\nJust want to test without making permanent changes? Use sysctl -w followed by the key=value.\nTip: Using sysctl you can tune your system and is a good way to harden the Linux kernel.\nEnable core dumps The primary reason to allow core dumps is for troubleshooting purposes. The dumped memory of the process can be used for debugging issues, usually by more experienced developers. A software vendor may ask to enable core dumps. Usually to discover why a process crashed in the first place and find the related routine that caused it.\nEnabling core dumps on Linux is similar to disabling them, except that a few specific details should be configured. For example, if you only need details from a particular program, you can use soft limits. This is done by using -S which indicates that it is a soft limit. The -c denotes the size of a core dump.\nulimit -S -c 0\nNext step is to only allow \u0026lsquo;my-program-to-troubleshoot\u0026rsquo; to create a core dump.\nulimit -S -c unlimited my-program-to-troubleshoot\nIf you want to allow all processes to use core dumps, use the line above without the program, or set a system limit in /etc/security/limits.conf .\nsoft core unlimited Troubleshoot setuid binaries Binaries that have a setuid bit set, can run with root permissions. This special type of access needs to be restricted as much as possible. Also for the creation of core dumps, it needs to be configured properly. This is done with the sysctl fs.suid_dumpable key.\nValue Meaning 0 Disabled 1 Enabled 2 Enabled with restrictions So if you like to troubleshoot programs with a setuid bit set, you can temporarily change the fs.suid_dumpable to 1 or 2. Setting it to 2 is preferred as this makes the core dumps only readable to the root user. This is a good alternative for systems with sensitive data. Setting the option to 1 is better suited for personal development systems.\nCreate normal dump files One of the big mysteries with Linux systems is where the core dumps are located. Linux has a trick in place to capture core dumps. This particular setting is done via the sysctl kernel.core_pattern setting or /proc/sys/kernel/core_pattern. Most systems will have a pipe (|) in this setting to indicate that a program needs to take care of the generated data. So if you wonder where your core dump goes, follow the pipe!\nCore dumps on Ubuntu systems are typically going to Apport. For Red Hat based systems it may be redirected to Automatic Bug Reporting Tool (ABRT).\nYou can temporarily change this setting, by echoing \u0026ldquo;core\u0026rdquo; to that file, or use the sysctl utility.\nsysctl -w kernel.core_pattern=core\nAn important note is that this change might not be enough. It depends also on your fs.suid_dumpable setting. A warning will be logged to your kernel logger if that is the case.\nSep 06 15:51:18 hardening kernel: Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.\nWhen needed set your core_pattern to a full path, optionally with variables defining who was running it, the PID, etc.\nsysctl -w kernel.core_pattern=/var/crash/core.%u.%e.%p\nIn this example, our dumps will contain the user id, program name, and process id.\nSystemd core dumps When using a modern Linux distribution you will most likely have systemd enabled. You might need to override settings via /etc/sysctl.d/50-coredump.conf and define how and where you want to store your core dumps.\nUsing systemd-coredump Your kernel.core_pattern may be defined to use the systemd-coredump utility. The default path where core dumps are stored is then in /var/lib/systemd/coredump.\nTesting your configuration Most other tutorials just give you the settings to be configured. But how would you know things work as expected? You will need to test it!\nCreate a core dump Option 1: Create an unstable program Let\u0026rsquo;s create a simple program. Its primary goal is to crash when being executed and then optionally create a core dump. Install gcc on your system and create a file crash.c in your home directory.\nint main() { return 1/0; } This program will start the main function and return an integer value (number). However, it is dividing 1 by zero, which is not allowed and will crash. The next step is compiling our little buggy program.\nOur unstable little program\nEven the compiler shows our program contains a serious issue and displays a warning about it. Now let\u0026rsquo;s run it and see if this is the case.\nA nice crash!\nFrom this single line, we can actually learn a few things. First of all that it quit with an exception, specifically referring to floating points. This is a decimal number format for programs, so it may indicate that something happened while doing some math. Another conclusion is that the core is dumped due to the (core dumped) addition at the end. If core dumps were disabled, this would not appear.\nGreat, so with this crash above we have now a dumped file, right? Not exactly. Depending on your Linux distribution things might not as simple as it looks. Each distribution deals differently with core dumps and the default settings. Most recent Linux distributions also use systemd now and the rules have slightly been changed with that as well. Depending on your configuration, you might need to search for your core dumps. So here are some tips to ensure everything is configured correctly.\nOption 2: kill a running process Instead of using a test program, you can also terminate an existing process. This is done by using the SIGSEGV, which is short for segmentation violation and also known as a segmentation fault.\nkill -s SIGSEGV PID\nIf you replace PID with \u0026ldquo;$$\u0026rdquo; the current program (most likely your shell) will crash. Everything for science, right?\nOption 3: using gdb If you have the developer debugging tool gdb installed, then attach to a process of choice using its process ID (PID).\ngdb -p 1234\nThen when at the gdb prompt, generate the core dump by invoking the generate-core-file instruction. After using this command, it should return you output.\nSaved corefile core.1234\nCheck ulimit settings The ulimit settings define what may happen when a program crashes. So it is safe to first check this, for both root and a normal non-privileged user.\nCheck hard limit for core dumps:\nulimit -H -c\nCheck soft limit as well:\nulimit -S -c\nCheck the core pattern Use the /proc file system to gather the value and change it temporarily during testing. If you prefer using sysctl, then query the kernel.core_pattern key.\ncat /proc/sys/kernel/core_pattern\nIt might show something like this:\n|/usr/share/apport/apport %p %s %c %P\nIn this case, a crash will be piped to the apport utility. So this means that crashes are going to be analyzed by Apport . Normally crashes are found in /var/crash, but may also be in /var/spool or /var/lib/systemd/coredump on other Linux distributions.\nCheck the journal (systemd) In our case journalctl shows our crash, so that\u0026rsquo;s a start.\nSep 06 15:19:23 hardening kernel: traps: crash[22832] trap divide error ip:4004e5 sp:7fff4c2fc650 error:0 in crash[400000+1000]\nAfter checking all these settings you should be able to create a nice core dump.\nConclusion Core dumps can be useful for troubleshooting, but a disaster for leaking sensitive data. Disable core dumps when possible, and only enable them when really needed. In such case check if the files are stored safely, so normal users can\u0026rsquo;t see the data. And independently of what choice you made, always test if your configuration does work exactly as you expect it to work.\nDo you have other tips regarding core dumps? Share them in the comments!\n","permalink":"https://linux-audit.com/software/understand-and-configure-core-dumps-work-on-linux/","tags":["core dump","software","sysctl","troubleshooting"],"title":"Understand and configure core dumps on Linux"},{"categories":["Network"],"contents":"Also wondering what particular files do on Linux? One of those files we recently rediscovered during auditing is the /etc/networks file. For some reason it was always there, yet we never change it.\nOutput of /etc/networks\nWhen looking at the man page of networks(5) we learn its purpose (almost instantly):\nIt translates between IP ranges and network names It is used for tools like netstat and route It only works on class A, B, or C networks It does not work on subnets Surprisingly enough a test with subnetting actually showed the right names during our test. Digging through the related system calls (getnetbyaddr(3), getnetbyname(3), getnetent(3)), it did not reveal the reason why it worked.\nQuery the networks database Linux systems use different databases which can be queried with the getent utility. It is the abbreviated name for \u0026ldquo;get entries\u0026rdquo; and uses the Nameserver Switch Services libraries. You may have been a package like libnss installed on your system. You can consider them wrappers around files and data structures, like the /etc/networks file.\nQuerying the networks database is simple:\ngetent networks\nDepending on your network configuration it may be more populated than our test system, which shows only one network. In this case just the \u0026ldquo;link-local\u0026rdquo; network (169.254.0.0), which is a reserved local network range for when no DHCP server is available. The system may then fall back on this range, to have at least some network.\nA lonely network\nConclusion The /etc/networks file has a limited purpose. It may help in displaying network names for some network related utilities. For more advanced network setups if may be useful to quickly show what network range belongs to which customer. We can conclude the /etc/networks file will not have a huge impact on how the system works.\nDid you configure your /etc/networks file, or use it in a particularly useful way? Let it know!\n","permalink":"https://linux-audit.com/the-purpose-of-etc-networks/","tags":["file system","linux","network"],"title":"The purpose of the /etc/networks file"},{"categories":["Linux"],"contents":"Every operating system needs memory to store program code segments and data. This is also true for Linux systems. The problem: there is a lot of information available regarding memory usage and its behavior. Let\u0026rsquo;s discover how Linux manages its memory and how we can gather memory information.\nAfter reading this guide, you will be able to:\nShow the total amount of memory Display all memory details Understand the details listed in /proc/meminfo Use tools like dmesg, dmidecode, free, and vmstat Linux memory information Random access memory When we talk about memory in this article, we usually mean random access memory (RAM ). This is the memory which can be used for both showing and storing data. Typically we will find in this type of memory the programs that are running on the system, including the Linux kernel itself. Besides the program code, memory also stores a lot of data. A good example is when you are running a MySQL database server. The program itself is relatively small, the data itself is huge. So we will also have a look at tuning programs and their memory usage, as this is typically a problem with memory-hungry programs.\nDetermine the amount of RAM The first step is to discover the amount of RAM we have in the system. There are a few ways on how to achieve this, starting from the data stored in dmesg.\ndmesg | grep -in mem\nThe output may look something like this:\nThis information shows the number of memory available in kilobytes. The first value shows what is currently available, the second value displays the total memory in the system. These values are usually very close. This indicates that most of the memory can be used and is a good thing. The small portion \u0026ldquo;missing\u0026rdquo; is used by the initial loading of the kernel. If there is a big gap, then this might be caused by the kernel and how many memory it can allocate. Especially with 32 bits versions of Linux, this number is limited.\nDetails and information about RAM modules The next step is learning more about the RAM modules itself. You will need the dmidecode utility for this, which is available for most Linux distributions. To gather memory information, tell the dmidecode to only show information for device type 17.\ndmidecode --type 17\nDepending on your hardware it may be able to extract the specifics of your modules and show detailed information. You may need to run this as root user. Normal users won\u0026rsquo;t have the right permissions to read all information.\nIn this output above you can see the details of the first memory module. We see it is a chip of 4 GB and configured at a 1600 MHz speed. This is a great way to determine the memory available in a Linux system, together with detailed output. Unfortunately, the command does not always play well with virtual systems.\nNo data is displayed on our virtual test system\nLet\u0026rsquo;s move on to the next set of utilities and gather details regarding memory usage.\nAvailable memory After the Linux kernel is booted, it is time to start programs. The kernel itself is not responsible for the programs. Instead, it delegates this responsibility to a service manager like init or systemd. This process is the first to be started and will get process ID 1. Its duty is to start other services and programs during the lifetime of the system. Each program will consume some amount of memory, depending on the program size and the related data. Let\u0026rsquo;s have a look at some ways to see available memory on Linux and retrieving related details.\nSee available memory with free command The first command to obtain available memory information is the perfectly named tool free.\nThis utility shows two different types of memory: normal memory and swap memory. Swap is a type of memory that you want to avoid needing as much as possible. If it would be used, then it means your normal memory is full. The system will then leverage the swap memory to temporarily store data, at the cost of disk operations. As they are much slower than normal RAM, your system will be impacted. In this screenshot, we see the swap is not used, which is good.\nThe free utility retrieves this memory information from a file named /proc/meminfo. Let\u0026rsquo;s have a look at that as well.\nDetails from /proc/meminfo The next step to obtain everything available regarding memory is found in the procfs tree, usually mounted under /proc. This file is very extensive, so have a look at it on your system:\ncat /proc/meminfo\nA partial output listing showing how memory is used\nMemory management under Linux is extensive and changed over time to what it is now. This results in a delicate system that optimizes memory usage as much as possible. Let\u0026rsquo;s get into some of these fields and understand better how Linux does its job.\nCached, SwapCached The system does a lot of repetition, including reading the same files or data. Everything that goes into memory and is no longer needed, will be kept for a little bit longer. If you then request the same data while it is in memory, you will get it almost instantly. This is what happens when you run a find command on a particular directory the first time, which usually takes a while. Run it again and it will be much quicker.\nActive, Inactive A page cache optimizes access to files. These buffers can be recently used (=active), or not (=inactive).\nActive is the total of Active(anon) and Active(file). Similarly, Inactive is the total of Inactive(anon) + Inactive(file).\nSwapTotal, SwapFree These provide insights in the configured swap memory and how much is left. Ideally, the SwapFree value is equal to SwapTotal, meaning no swap is in use at that time. Swapping is disk intensive.\nDirty The Dirty field refers to data that is stored in memory and still needs to be written to the disk.\nYou can test this easily by writing to a temporary file and compare the value before and after.\ncat /proc/meminfo | grep Dirty \u0026amp;\u0026amp; dd if=/dev/zero of=/tmp/testfile.txt bs=1M count=10 \u0026amp;\u0026amp; cat /proc/meminfo | grep Dirty\nNote: if you repeat this command, you will see the effect of smart memory management. In that case, the value before and after will most likely be the same, as some data was cached and directly returned as a finished action. You can counter this by retrieving random data from /dev/random .\nSlab, SReclaimable, SUnreclaim The kernel does a lot of repetition during the time it is running. Some objects, like asking for the specific inode of a file may be performed thousand times a day. In such case, it would be wise to store it in a quick reference list, or cache. Slab is the combination of caches for kernel objects, to optimize those activities that happen the most.\nThe Slab field is the total of SReclaimable and SUnreclaim.\nSlab: 32272 kB\nSReclaimable: 18144 kB\nSUnreclaim: 14128 kB\nNFS_Unstable For systems that use NFS this is a good measurement to see how much data is not committed to the storage yet. For systems without NFS, this value can be ignored and is usually just zero.\nMore fields If you compared these fields with your own system, you will discover there are more fields. Depending on your workload, you will have to discover what fields make sense to monitor. What does generally work well during troubleshooting, is comparing similar systems and check for the differences in /proc/meminfo. It may give a good indication of where memory is used and what keeps the system busy.\nUsing vmstat utility Another nice utility that is often available is vmstat. With -s we can query memory statistics.\nWe can also query the previous mentioned slabs.\nMonitoring memory usage in Linux If you have a monitoring system in place, then two key attributes from /proc/meminfo should be monitored.\nMemFree SwapFree By monitoring these two values you may discover memory leaks and badly optimized systems. You may also pick up on a misbehaving process now and then.\nFor environments that have many similar systems with a typical workload (e.g. lots of web servers doing the same), then it would make sense to monitor more of the keys in /proc/meminfo. Storing the data a few times per day may give you the ability to compare systems and find exceptional events.\nFrequently Asked Questions How can I find out the total physical memory (RAM)? Use the free command to show the total amount of memory and how it is assigned.\nWhat Linux tool can I use to see the details of the memory modules?\nUse the hwinfo tool to gather details regarding the memory. If that is not available, then consider using the output from dmesg.\nhwinfo --memory\nHow can I see which processes consume the most memory? Use the ps command to show memory usage and do a reverse sort.\nps -e -orss=,args= | sort -nr | head\nWhy are the buffer and cache use so much memory on Linux? Linux considers unused memory to be wasted memory. So it will use as much memory as possible to speed up the performance on the system. The related caches and buffers contain typically data related to the file system. That is also why a second run of the find command in the same directory runs much quicker. The kernel will reassign memory to processes when needed.\nWhere does the kernel store the details regarding virtual memory management? Most of the related settings, like how to act during an Out-of-Memory event, will be stored in /proc/sys/vm.\nConclusion Linux memory management is an extensive subject and there is a lot to learn. Make sure to understand the basics, like how to obtain memory information, including that of RAM and swap. This is of great help during troubleshooting and to know what programs need to do their job.\nDid you learn something from this article? Great! Share it on your favorite website or with others. If you have a nice tool for memory analysis or got a question, let it know!\n","permalink":"https://linux-audit.com/understanding-memory-information-on-linux-systems/","tags":["forensics","guide","memory","proc","processes","procfs","ram","swap","system administration"],"title":"Understanding memory information on Linux systems"},{"categories":["Kernel"],"contents":"Configuring ASLR with randomize_va_space The Linux kernel has a defense mechanism named address space layout randomization (ASLR). This setting is tunable with the randomize_va_space setting. Before making changes to this setting, it is good to understand what this Linux security measure actually does and how it works.\nUnderstanding ASLR In 2001 the term ASLR was first introduced as a patch to the Linux kernel. Its main goal was to randomize memory segments to make abuse by malicious programs harder. A normal program consists of several components, which are loaded into memory and flagged with special properties. Some pieces of the program are executable bits, others are normal data. Before going into these properties, let\u0026rsquo;s first determine the main goal of a program. Simply said, it should have a start procedure, maintain itself, and finally end. For some programs this whole cycle can take milliseconds, others may take years to complete. It all depends on the program, its stability and how often a system is rebooted.\nGuarding against malicious software attacks While the program runs in memory, we want it to be protected against more evil programs. One of the tricks they use is hijacking the stack pointer. This is a like a traffic agent stating where to go next. Evil programs want to abuse this and perform a redirection trick to insert malicious code into a running program. For this reason, programs have different sections and should be properly flagged in the memory. A section where only normal data is stored should be marked as non-executable. Executable code that does not dynamically change, should be flagged as read-only, etc.\nMemory randomization Besides the mentioned protection mechanisms, we can add another layer and defend against memory misuse. This layer is randomization of virtual address space. For this to work, the binaries running on the system should be a position-independent executable. This means it does not require static memory addresses to fulfill its duties. Since many years this feature is common, which enabled the kernel to apply memory randomization.\nDefault randomize_va_space setting Modern Linux kernels have ASLR enabled by default with the specific value 2.\nNormally you might expect a value of 0 (disabled), or 1 (enabled). In the case of the randomize_va_space setting, this is true as well. When setting the value to 1, address space is randomized. This includes the positions of the stack itself, virtual dynamic shared object (VDSO) page, and shared memory regions. Setting the option to value 2 will be similar to 1, and add data segments as well. For most systems, this setting is the default and the most secure setting.\nNote: some older platforms do not support this setting.\nConfigure randomize_va_space If you want to change the value, it can be done temporarily or permanently. One option is to set the value via the pseudo proc file system. You need to be root to change this setting.\necho 2 \u0026gt; /proc/sys/kernel/randomize_va_space\nAnother option to temporarily change the setting is via the sysctl command.\nsysctl -w kernel.randomize_va_space=2\nTo make this setting permanent and active after a system reboot, add the option to /etc/sysctl.conf.\nNote: kernel hardening is a good way to improve the security defenses of a Linux system. To perform a security scan in this area, you can use the open source tool Lynis. It includes the scanning of kernel settings and sees if they are already tuned.\nDisable ASLR If you temporarily want to disable, you can set the value to zero with the instructions above. If you just want to test for a single program you can use the setarch command. This leverages a so-called personality flag.\nsetarch \\uname -m` -R /root/mybinary`\nThe -R option disables the randomization of the virtual address space by turning on ADDR_NO_RANDOMIZE. This option allows programs to disable ASLR and run without any randomization.\nIf you are doing research (e.g. to test how buffer overflows work), keep in mind that also the compiler has some protection mechanisms in place. If you found this page to achieve that, then use this disable stack protection during compilation.\ngcc -fno-stack-protector -z execstack -o program program.c\nMore tunables via sysctl Are you looking to perform additional kernel hardening of your Linux system? Have a look at the knowledge base section sysctl on the Linux Security Expert website.\n","permalink":"https://linux-audit.com/linux-aslr-and-kernelrandomize_va_space-setting/","tags":["kernel hardening","memory","programming","sysctl"],"title":"Linux and ASLR: kernel/randomize_va_space"},{"categories":["Interviews"],"contents":"Linux malware, research, and more With great pleasure, we interviewed unixfreaxjp. He is the leader and founder of the malware research group MalwareMustDie . We want to learn about their activities, Linux malware, and useful skills for security professionals. Keep reading!\nInterview MalwareMustDie About the MalwareMustDie organization So for those never heard about MalwareMustDie, can you tell us who you are?\nAs stated on our web site. MalwareMustDie, is a white-hat anti cybercrime security research workgroup. launched in August 2012, is an Non Profit Organization media for security professionals and researchers gathered to form the work-flow to reduce malware infection in internet. We work to raise malware awareness by sharing general information of malware infection scheme and trend to the common users, helping security vendors and public automation malware-related scanning/decoding tools by providing in depth decode analysis to the recent malware infection frameworks, and work with legal authorities to take down malware domains, and its further threat intelligence.\nWe aim to establish good relationship vertically with authorities, and horizontally with the fellow researchers and security entities, so that cooperation can be enlisted in dismantling domains that host malware and its infectors in internet.\nWhy do work for free? What is there to gain?\nWe work as non profit “organization”, hardly can be called as a company. All of us are mostly employees or engineers with the day-work duties related with the network and internet administration or security profession. The organization is not receiving any income and costs were paid by the involved member’s own money on operations.\nCan you tell us a little bit about the MMD team? How big is it?\nWe maintained a steady value of members around 30 members, with the supporters included, it is around 60 people right now.\nLinux malware We typically hear that there aren’t viruses for Linux. Seeing the samples collected on websites like Packet Storm Security we know there is quite some malicious software around. What type of malware do you encounter?\nMalware or virus in Linux exists for a long time. In 10 years ago, it is not as popular as Windows malware. But things had changed since 2012 when the abuse of Linux infection through unattended Linux devices has started. And we have a complete series of types since then, from backdoors, rootkit, hacking tools (scanner/bruter etc), spam tools, exploit distribution tool, ransomware, botnet kits (via irc or etc protocols) and to the traffic DoS attack malware tool.\nDo you see any trend that suggests malicious software on Linux is increasing during the last years? What about ransomware for Linux, does that show up now?\nIn each time security community announces a new linux’s (or unix) services related vulnerability, the linux malware trend and infection is raising. In example: During the shellshock, The PMA (phpMyAdmin) vulnerabilities, Apache Struts vulnerabilities, various OpenSSL vulnerabilities that leads to illegal authentication, and now the IoT’s factory credential setting flaw, all of these is (was) raising the Linux malware infection and distribution to the affected systems.\nRansomware is in “a boom” in cyber crime business. There are various type of Linux ransomware that encrypt websites data or the server’s data now, since to code an encoder or encrypter program is not difficult at all. For the cyber crime, ransomware is always high in profit and low in risk compares to the in real life extortion or ransom crime, most of the professional cyber mafia are on this “business” now.\nThe thing is that Linux is based on open source, dissecting ransomware in Linux is only a matter of time. You just can not mess with Linux/UNIX system administrators, for years they are the one who ready with backups, images and more savvy solution to prevent any of their services go down.\nWhen someone finds a piece of malware on their machine, they can upload it on your website. What happens with the samples?\nWe just analyzed each samples, each one of it. Then we checked whether the protection layer i.e. antiviruses or etc signature (IDS, Yara, etc) already cover it, then we go deeper to the uncovered ones. When it comes to an unknown malware and it is aiming public level of threat then we post the awareness in our blog. Sensitive cases like APT for example, we don’t expose at all.\nRecent developments You are known on Twitter as @malwaremustdie. You are using a lot of crusader pictures. Has it to do with religion, or is it something else?\nIt is just a symbol, just as Linux uses Penguin and FreeBSD uses a Daemon, we use knight images during the medieval era. No it is not related to any religion at all but all of the members are religious and decent citizens. The “Crusade” term is also symbolizing the hard effort we face to fight malware and crime scene behind it, it is a big deal, knowing that the malware still exist for, more than 20 years now.\nLast year the Twitter handle became a private account. You also announced a lot of people to be removed from the followers. What was that about?\nWe are not active anymore on twitter. It’s all about security. There were malware people are lurking us. @malwaremustdie had 15,000+ followers and now we have around 1,300 after I reduce them. Most of the followers are the blackhat lurkers. They learned from what we tweet and use the information to improve their malware, some blackhats are using the vulnerability that we found to improve their malware too.\nThese lurkers are using the predicate as “security enthusiast”, “malware researcher”, “reverse engineer students”, “system administrator” and some of them are even faking real researcher’s pictures, names or avatars that they stole from respectful researchers from other SNS. In order to avoid this, to the people that we don’t know, we vetted and asked followers to inform about them self. We disconnect the follower who doesn’t explain. But our direct message is always open for them who want to re-follow after they give more details about them self.\nI also run several scripts connected via twitter API to check the validity of accounts who tried to follow us, if the indicator is RED we won’t even answer to the request. Right now we have almost 500 requests already, that was still flagged as RED. We need to conduct our research peacefully and to OPSEC our comm better, right now we are in the most happiest state.\nBut people can still read the blog and learn about the details, right?\nYes, blog is the recommended ways for the public, including the malware bad guys, to read. The information in the blog was filtered, we passed all of the necessary details to the law enforcement before or during the time we blog it now.\nIf someone interested in malware and security, do they make a chance of being accepted as a new follower?\nWe are done with Twitter, if you refer to it. Right now I am not willing to add twitter followers anymore. People can follow us via blog or IOC feed we released, journalism and legit researchers know exactly where to reach us.\nOur twitter DM in twitter is open to anyone. To ask questions or for an introduction. The funny thing is, blackhats are using this channel a lot to send “their messages” etc, instead of whitehats.\nProfessional skills Is there a benefit for security professionals to learn analyzing malware samples? How could you use it in your daily work?\nIt is important for security professionals to know how to check a malware sample. They don’t have to reverse engineer the sample, but to identify it as malware up to some level. Using a tool to detect a malware is okay for the novice computer users, and they will go to the security professional to ask more issue or problem. Knowing by yourself about a malware sample will always bring more good than bad. Please, always conduct your analysis in the very safe environment.\nIf one would start with malware analysis, what would be a good way to start? Any resources we need to know about?\nThe internet is the best resource for learning in this era. This is why the internet has to be kept clean for all of us to learn and communicate on the safe services. There are a lot of reference for malware analysis on the internet. For Windows malware, I suggest you to take a look on fumalwareanalysis.blogspot.com , and for Linux malware, our blog at blog.malwaremustdie.org is very rich for learning in real cases.\nWhat makes that someone who analyzes malware go from being good to a great researcher?\nI don’t know how to answer this. Personally, I never consider myself as malware researcher at all. I myself is as UNIX system engineer and protocol filtration developer. I like C programming in profession and reverse engineering as troubleshooting method and a hobby, and the best object to reverse is malware.\nMany researcher “experts” laugh at the way I analyze a binary, as they think I go into too much detail of the binary I check. One thing that I do when I analyze a malware is, be relaxed and don’t get excited too much, I have to stick with the binary in hand and I have to extend my skill set if we face something I don’t understand. I like to understand what every opcode means, how is this binary was compiled in such way, how it is packed, how it is executed, etc. These are raising some “Why?” questions that I like to solve myself. The more you solve the better understanding you get from that malware.\nDealing with malware is not as same as we deal with legit software. Malware is coded with “lies” intact. But binary file never lies, they always try to tell you more of the badness inside of its bytes, and you just have to improve yourself to listen to it.\nThanks for your time and answering these questions. One more question: which security person do you think we should interview and why?\nI suggest to interview Linus Torvalds, seriously, for his perception is very important for the roadmap of Linux kernel that is recently abused much by many malicious efforts. Ask questions about what he can develop in the future to make Linux kernel more secure than now.\nDid you like this interview? Share it on your social network.\n","permalink":"https://linux-audit.com/malware/interview-malwaremustdie-linux-malware-research/","tags":["linux","malware","professional skills","ransomware"],"title":"Interview: MalwareMustDie and their Linux malware research"},{"categories":["Crypto"],"contents":"Everyone has secrets. Or at least some data you don\u0026rsquo;t want to show others, right? Vim is a common editor to be found on Linux systems. It has an option to create and use encrypted files. We will look at how to configure it and use this encryption capability.\nEncryption is the process of fiddling with data so that others no longer can\u0026rsquo;t read it. The idea is that you still can, so when we speak about encryption, we can\u0026rsquo;t ignore decryption. This also means that we need a good cryptographic algorithm. This way we can store our original file into an encrypted version. Then when we later need the data again, we can decrypt it.\nImportant to notice is that the implementation of encryption in vim is suitable for personal use. If you want to protect intellectual property, trade secrets, or even more sensitive data, consider other options. We will discuss these later.\nConfigure vim To use encryption, we first need the right support in your vim installation. Secondly, some configuration is required to activate the right settings and doing it securely.\nCryptv support No cryptography magic will happen if we don\u0026rsquo;t have the right support. We need the cryptv support compiled into vim.\nvim -version\nThis output should give you version details and related capabilities. Search for +cryptv in the output.\nBlowfish2 support Your version needs to be at least 7.4 with patch level 401. If your Linux distribution ships an older version, you can only use \u0026lsquo;blowfish\u0026rsquo;. The implementation of blowfish in vim is incorrectly implemented, resulting in weakened encryption. This makes it possible to crack the first 64 bytes of the file and possibly more.\nSet encryption method We start by setting the encryption method we want to use.\n:set cryptmethod=blowfish2\nTip: you can also use cm as an abbreviated version.\nDisable backups During editing your files you may not want to leak any sensitive data. Backup files have the main purpose to make a copy of your data, but that is not what you may want in this case. Disable the creation of these files.\nDo not make a backup\n:set nobackup\nDo not write to a temporary file first\n:set nowritebackup\nIf you still prefer to have some backup files, you could enforce writing temporary files in directories you control and clean those out at your convenience. This way you still have the backup, with slightly more control over where any sensitive data may be located.\n:set backupdir=/vimtmp,.\n:set directory=/vimtmp,.\nAnother tweak to still allow backup files is disabling them for some specific directories\n:set backupskip=/tmp/*,/private/tmp/*\nDisable viminfo The viminfo file also maintains information about your vim sessions. As this may contain sensitive data, disable the file if you don\u0026rsquo;t want to take any risk of leaking data.\n:set viminfo=\nDisable swap The creation of swap files (.swp) can be disabled as well.\n:set noswapfile\nIf you want to reuse these settings, simply add it to your ~/.vimrc file. For example:\nset cryptmethod=blowfish2\nset nobackup\nset nowritebackup\nset viminfo=\nWith these settings in place, we can start using the encryption options vim has to offer.\nEnable encryption of a file Start vim with the -x option.\nvim -x mynewfile.txt\nFor a file that is already opened, use the :X option and vim will ask you for an encryption key. This will be used to mangle all data and ensure others (without the key) can\u0026rsquo;t see the data.\nAfter saving (with :w) the file is stored on disk. You can validate that the data is encrypted by using the file command.\nWhen opening up the file you will be asked for your encryption key. If that matches the one you provided before, the file is editable again.\n","permalink":"https://linux-audit.com/using-encrypted-documents-with-vim/","tags":["cryptography","encryption"],"title":"Using encrypted documents with vim"},{"categories":null,"contents":"Using file flags on macOS While performing system hardening on macOS, you may encounter a typical chmod error. Something like this:\nchmod: Unable to change file mode on /usr/bin/gcc: Operation not permitted\nEven with root permissions, you can\u0026rsquo;t change the permissions of some files. How is this possible? This is caused by flags.\nShowing file permissions and flags To see if a file has any flags set, use the ls command with the l (el) and O (capital o).\nls -lO /usr/bin/gcc\nThis will show if the file is immutable, or any other file characteristics.\nChanging flags on files If you want to change the permissions of a file, you first need to turn off the related immutable flag.\nchflags nouchg /usr/bin/gcc\nNext step is changing the permissions.\nchmod 750 /usr/bin/gcc\nThen turn on the immutable flag again.\nchflags uchg /usr/bin/gcc\nSee man chflags for more details about flags.\n","permalink":"https://linux-audit.com/changing-file-permissions-on-macos-and-using-flags/","tags":["file permissions"],"title":"Changing file permissions on macOS (and using flags)"},{"categories":["Network"],"contents":"The /etc/hosts file is one of the few files you will always find on a Linux system. It stores the \u0026lsquo;hosts\u0026rsquo; database, and can be used to resolve between IP addresses and hostnames. Although the file is very simple structured, it is still common to see minor issues with name resolving on systems. Guess what, your /etc/hosts file might be causing more trouble than you think. A regular check up won\u0026rsquo;t hurt.\nOrder matters in name resolving Linux systems have a library called NSS . It defines the databases for resolving between identities and names. Not just between hostnames and IP addresses, but also your user accounts, protocols, and services. Knowing a little bit more about this library is good to know, especially when working both IPv4 and IPv6 addresses and the combination of local files and DNS.\nThe first step is knowing how the system determines where to look when it wants to do some name resolving. We use the /etc/nsswitch.conf file for this.\nLet\u0026rsquo;s have a look at the file:\nThis file shows for each database type what order it will use and what specific query mechanism. For our hosts database we see it queries first the applicable files, then uses DNS. This is good to know, as we can leverage to overrule hardening with the /etc/hosts file.\nPlease note that some tools will not adhere to this order. Tools like host and dig are meant to query data via DNS. Querying the \u0026lsquo;hardening\u0026rsquo; hostname, which is listed in /etc/hosts, will still result in an error:\nHost hardening not found: 3(NXDOMAIN)\nIf you want to validate how resolving is performed by a tool, use the strace command:\nstrace host test\nGive it a go and see if you can find /etc/hosts in your output (we can\u0026rsquo;t).\nThe hosts database The hosts database is formed by the /etc/hosts file. We can query it in two ways:\nView /etc/hosts Use getent The first option is easy, as we can use the cat command for that.\ncat /etc/hosts\nAnother command is getent, which might be less familiar, yet often available by default.\ngetent hosts\nBoth will query (or show) what is in the hosts database. While this might have the same goal, the output might surprisingly be different.\nIn this screenshot we see the last two lines missing from the getent command output. This is because they are not normal hosts. They can be compared to broadcast addresses in IPv4, with ip6-allnodes for all systems (including routers), the ip6-allrouters for just the routers in the network segment.\nWhat can be wrong with /etc/hosts? Although the /etc/hosts file is a simple structured file, there are a few things that should be checked. So next time you are on a system, become an IT auditor and check the following parts:\nYour hosts file is not a DNS replacement If you have added more than 10 systems to your /etc/hosts file you may consider moving that to a separate DNS zone. Even if it is for internal usage, name translation is perfectly performed by DNS. Add your internal zone to your name servers for optimal caching and easy of management. Even if you use configuration management tools like Ansible or Puppet, your host file should not be storing many entries.\nUsing your local hosts files may also have the risk of introducing unexpected behavior when some system name is reused for example. If you truly only use some entries for temporarily testing, then the hosts file can be a great option. From experience we can say that temporary often results in permanent (on purpose, but more often by accident). So try to be disciplined and avoid changing the hosts file and keep your systems tidy.\nStill want to maintain a small list of DNS records and offer those within the internal network? Consider running dnsmasq and use that as a resolver on other systems. This way you have to maintain the DNS entries in just one place.\nNo FQDN provided Still many Linux installations are not properly configured when it comes to the domain name. Even if you provided it during the installation process, it may not have been propagated to the /etc/hosts file. The FQDN defines the full hostname, including a domain.\nYou can easily check if your Linux system is properly configured by using the hostname command.\nhostname -d\nNo output of this command means there is not hostname configured. In that case change your hosts file into this format\n[IP address] [FQDN] [hostname]\nOne typical error is that the last two are reversed. The longest match, which is the FQDN , should be at the front to get it working.\nLocalhost mapping The localhost entry defines the local system. It should always map against 127.0.0.1 or ::1 (IPv6) to prevent issues. To check this, use the getent utility.\ngetent hosts localhost\nThe other way around should be return at least localhost, with optional some aliases.\ngetent hosts 127.0.0.1\nConclusion The /etc/hosts file is used on Linux to support local name resolving. The file itself should remain as small as possible, so the remaining entries can do their job. One of them is resolving the localhost entry back to 127.0.0.1 (IPv4) or ::1 (IPv6). The other purpose is to define the domain name of the system, to properly form the fully qualified domain name (FQDN).\nWant to test these things automatically for all systems? Then check out Lynis, as it has built-in tests for that (and much more).\n","permalink":"https://linux-audit.com/is-your-etc-hosts-file-healthy/","tags":["dns","file integrity","hostname"],"title":"Is your /etc/hosts file healthy?"},{"categories":["Software"],"contents":"The question about what the differences are between rkhunter and Lynis is showing up more and more. Time to share the purpose of both and show the difference in its usage. As the author of both tools, I should have done this nine years ago. So with some little delay, here it is.\nRootkit Hunter Written in 2003, rkhunter had the goal to detect malware on Linux and UNIX-based systems. The main target was rootkits, with an occasional detection mechanism for a common backdoor. The secondary target was promoting a few best practices, like disabling direct root logins via SSH.\nThe rkhunter tool is written in shell script to allow portability and support more than just Linux systems.\nLynis Lynis was created in 2007, also as a set of shell scripts. Where rkhunter focuses on malware, Lynis takes a more generic approach. The primary goal is to provide tips for system hardening. It does so by detecting weak configurations, search for vulnerable software packages, and looking at several system characteristics. These include the processes that run or some files that may be present. Depending on the outcome of those, more tests will be executed.\nThe output of Lynis looks slightly similar to what rkhunter uses. This is because of some screen routines used to share any findings.\nWhich one should I use? The primary difference between the two tools is that Rootkit Hunter focuses on malware detection, Lynis on performing a security assessment. For that reason, you should at least use Lynis, combined with a malware scanner. That could be rkhunter, ClamAV, LMD, or one of the commercial solutions. It mainly depends on what kind of malware could possibly reside on that particular system. A web server with file uploads has different threats than a mail server.\n","permalink":"https://linux-audit.com/tools-compared-rkhunter-vs-lynis/","tags":["comparison","lynis","rootkit"],"title":"Tools compared: rkhunter VS Lynis"},{"categories":["Software","Software Development"],"contents":"While \u0026lsquo;shopping\u0026rsquo; for some libraries, it struck me how many open source software projects are suffering from basic mistakes. Well, mistakes might sound too harsh. What I mean are those things you find on a project, which could be better. They are usually things not considered by the developer, as we (developers) were never told about them.\nDoing 20+ years of open source development now, I can safely say I made many mistakes. Time to get them all fixed and document them, part of the open source community. I\u0026rsquo;m Michael Boelen , and you may know some of my work, like Rootkit Hunter (rkhunter) and Lynis. Here are some of the lessons I learned. You can use them next time when choosing a new open source project and make a better judgment call. If you are developer, then you can use these lessons to improve your own project.\nYour website or project looks outdated A first impression is everything, especially when it comes to new software. Your website looks outdated or has a copyright message on it of four years ago. It tells me the website is not maintained, so it makes me wonder if the software is any better.\nSpeaking about outdated software, this is a serious killer for getting new users on board. Even if you don\u0026rsquo;t do many updates, do release a small version now and then. This way we know the project is still alive.\nKeep it up-to-date When doing development on rkhunter, I had moments in which I did not have the time to work on the tool. And sometimes I simply had no energy to do so. The result was a tool that lacked updates, a lot of questions in my mailbox, and people asking for updates. To combat this, I handed over development to a group of people and retracted. That is an interesting subject in itself. But what the most important thing at the moment was for me is the continuous development of the project.\nAfter I quit the rkhunter development, I started the security tool Lynis in 2007. For a long time, I could release versions. That is until my normal work got in the way. Long days of work and the development of Lynis came to a halt. With the lack of updates, people assumed the project was dead, which is also the rumor spread over several forums. The big lesson here is that people make assumptions, just by looking at a date. So one takeaway here is that it is better to release small and often, instead of releasing big updates.\nFirst impression tips Avoid old-looking websites like Sourceforge Get up a simple website, that is quick and easy to read Show me buttons with actions to take (like a download button) Display the release date of versions Set a three-month reminder to determine if a new version should be released I simply don\u0026rsquo;t get it You listed all the nice features your project has. What you forgot is to explain me why it matters and what problem you solve. And while we are it, I\u0026rsquo;m not even sure if I\u0026rsquo;m your target audience.\nAdopted the power of story telling Only a while ago I learned about the power of stories. Humans are programmed to tell stories, so we can relate it to our own experiences and share it with others. The Rootkit Hunter project never had a good story. The message was only about what it did. Things like the why and how were missing. That is something we fixed for Lynis obviously. While the perfect pitch always remains difficult, we have a better story now. And especially first time, we keep it simple in conversations. We compare Lynis with a health check at the doctor, or your the check your cars gets. Obviously then for Linux and specifically for security issues.\nTips for a better story Explain who commonly uses it Tell the benefits first, then list the features Show technical requirements of the person needed to use the software Ask your most active users to write a quick testimonial Compile stuff Please do not make me compile things. Yes, one day I have enough time to fully customize and make the perfect customized build, but not now. I still need to test your project and I want to do that quickly and see if we are a match.\nMake it easy to use Both projects are written in shell script. This helped a lot in getting it easily to work for others, as installation was not a requirement. It also avoids out-of-date software, as the latest version can easily be cloned from GitHub. Still, some of the users are not really familiar with permissions, and especially what the umask does. In Lynis a permission check is build in now, to ensure every new user is setting it up with the right ownership and permissions.\nSimplicity tips Provide a package or pre-compiled version Have a page with installation tips, but keep it small and simple Create a small movie Can\u0026rsquo;t get it to work out of the box So I installed your project and it refuses to work. It doesn\u0026rsquo;t start, or it doesn\u0026rsquo;t do what I expected it to do. Maybe it is my fault, maybe it is not. The error messages are vague and I have no idea what to do next. Well, I actually do: throw away the temporary directory I created. Don\u0026rsquo;t make me feel stupid and make sure that I have at least some results. Even if it is just a basic \u0026ldquo;Hello world\u0026rdquo; type of achievement.\nExperience with users and skill levels During the years I learned that there are two different types of users. Some do read the documentation first, then start using the software. Others take the reverse path and learn along the way. When you add skill level to these two options, things are more interesting. You will find that a lot of difficulties experienced by users never become visible to you as a developer.\nWith the rkhunter project, I learned that the audience was highly technical, directly from the start. Some of these early adopters even provided patches and feedback. But after a while that started to change. A more broad audience picked up on the tool. My mailbox filled up with questions like \u0026ldquo;How do I use this software?\u0026rdquo;.\nAlthough there was a README available in the tarball of rkhunter, it was too technical and too long. Splitting up the documentation into different stages of usage would have been much better. We have taken this step with the Lynis project and the results are good. While it is hard to measure if every user can quickly finds what he or she needs, we know things have been improved. By splitting up the documentation we save people from searching around in the document. This is because they now directly get to the right document when using a search engine.\nTips for first use Have a starter guide to help first-time users of the software When I start the tool without any parameters, tell me what I can use Have -h and --help as valid options and show me help instead of an error Ask your users how they experienced the installation and first usage Learn what documentation is visited often and keep improving it Software was advertised incorrectly Alright, I tried your software. Then I found out it was totally different than what I expected it to be. I removed it and most likely will never try it again.\nFocus on your audience No single tool can solve the problems for all of us. During the years of software development I learned that focus is important to achieve results, but also to attract the right type of users. Even in our business we use this principle by learning about our potential customers. If we believe someone is not a good fit, we will tell so and point them to another tool or even a competitor.\nTips for getting the right audience Include screenshots Show an instruction video to point out the main benefits Have a clear description what the user can achieve with your project Couldn\u0026rsquo;t find the license Choosing the right software license is not easy. Especially if you just want to share the thing you created with the world and don\u0026rsquo;t care about how it is used. By making no clear choice, it is hard for others to use it.\nTips for selecting the right license Pick the license that is close to your users (e.g. BSD for projects used on *BSD systems) Consider what it means to you if others would earn money with your code No mention of the author You might have created a nice piece of software. What I still don\u0026rsquo;t know is who you are. You as the individual who created the software, or the team. You don\u0026rsquo;t have to put everything up. At least your name and why you created this software would make me feel much better. We might have a lot in common and I might even want to get in touch with you, liking sending a \u0026rsquo;thanks\u0026rsquo; message.\nThe power of open source Many opportunities I had were due to my work on rkhunter. From getting a new job, a consultancy gig, and followers on Twitter and LinkedIn. I also received many books as a gift.\nTips for personal achievement Linked your name to the project Think about the possibilities open source contributions bring to your CV Mention the project on your LinkedIn and other social media Bonus step: promote your open source project If you got all of these points covered, then it is time to take the next step. This can be achieved with the right way to promote your open source project and get more users. After all you wrote your code to help solving a problem, right? Make it count and get as much people to learn about the problem and solution. You may gain not only more users, but also contributors.\nWhat mistakes did you make in your open source project? How did you solve it? Love to hear!\n","permalink":"https://linux-audit.com/software/why-we-use-your-open-source-project-or-not/","tags":["development","open source","software development"],"title":"Why we use your open source project (or not)"},{"categories":["Linux","System Administration"],"contents":"Determine Oracle Linux version Oracle Linux is based on Red Hat Enterprise Linux. At first, it may be confusing to determine what specific operating system is running. This is because both have the /etc/redhat-release file.\nIf that file exists, use the cat command to display the contents. Next step is to determine if there is a /etc/oracle-release file as well. If so, then you can be sure that Oracle Linux is running.\ncat /etc/oracle-release\nSample output might be: Oracle Linux Server release 6.7\nOther options Next time when you are on a system and not sure what it is running, use this:\nls -l /etc/*-release\nThis shows you any files that might give a hint on the operating system version.\nRelated files /etc/enterprise-release (older versions of OEL) /etc/issue /etc/issue.net /etc/lsb-release ","permalink":"https://linux-audit.com/how-to-see-version-of-oracle-linux/","tags":["how-to","linux","oracle"],"title":"How to see the version of Oracle Linux"},{"categories":["Hardening","Passwords","System Administration"],"contents":"The system hardening process of a system is critical during and after installation. It helps the system to perform its duties properly. This blog post shows you several tips for Ubuntu system hardening. It will dive into the most critical steps to take first. Then more specific hardening steps can be added on top of these. As most security guides only tell you what to do, we will also go into more detail on why a specific security measure is important. This way you can make educated decisions on what steps you want to do, or the ones to skip. After all, each system is different.\nSupported operating systems:\nUbuntu 20.04 LTS desktop and server Ubuntu 22.04 LTS desktop and server Ubuntu 24.04 LTS desktop and server Most of the steps will work on Ubuntu versions before and after these releases. This guide will cover both the desktop and server versions.\nUbuntu is already secure, right? Every Linux distribution needs to make a compromise between functionality, performance, and security. While Ubuntu has secure defaults, it still needs tuning to the type of usage. Ubuntu desktops and servers need to be configured to improve the security defenses to an optimal level.\nSystem hardening If you are new to system hardening, let\u0026rsquo;s start with a definition:\nSystem hardening is a technical process of increasing the security of a Linux system by reducing its attack surface. Those items that pose the most risk to the system are adjusted by taking specific security measures. This can be done by adding, adjusting, or removing certain components of the Linux system.\nDuring this steps in this guide, we will apply a combination of measures to improve the security of your Ubuntu installation and configuration. Although there are some specific Ubuntu security features, most of the hardening tips can be universally applied to other Linux distributions.\nDesktop versus server The process of Ubuntu system hardening is very similar for desktops and servers. The only difference is the purpose of the machine. Desktop users are most likely using it for browsing the web or reading emails. Privacy might be an important focus area. These activities are less likely on servers, where typically data integrity and availability are more important. So while the hardening steps are similar, keep always the role of the system in mind.\nHardening steps during installation The first few hardening improvements can be done during the installation. If you already have a system running, the most likely these steps can\u0026rsquo;t be easily applied after the fact. Consider reinstalling the system, or use the hardening measures as part of a future installation.\nUse strong passwords After the first installation steps, the creation of a user account is performed. This user will be added to the administrative group, allowing him or her to become root. For this reason, the password should be a strong password.\nA bad start of server hardening. Instead, use strong passwords.\nWhy a strong password matters: weak passwords don\u0026rsquo;t belong on systems. Not during development and especially not for production purposes. This is a serious risk as automated tools can perform many guesses per second, often discover weak passwords in just a few seconds. So system hardening should also apply to the strength of your passwords.\nTips to enhance your password: use longer passwords to make brute force password guessing much harder. One trick that is simple and powerful is adding a single character many times to your password (e.g. add 10 dollar signs at the beginning). Besides increasing the length, the variety of used characters is important. Add capitals, numbers, and other characters.\nUse disk encryption Enable encrypted LVM volumes during the installation of your Ubuntu desktop or server system. It is a great measure to hardening the system and data in particular. Although it won\u0026rsquo;t protect against all attacks, it matters for what we call data at rest. This means that when your system would get stolen, the data can only be retrieved if the attacker has the related key or passphrase to decrypt the data.\nSelect the guided partition method with \u0026ldquo;use entire disk and set up encrypted LVM\u0026rdquo;.\nNext step is selecting a passphrase. This is used during the boot process, to unlock the disk (or volume).\nMake it a good passphrase: longer is better\nWhy disk encryption matters?: Your system may be stolen, even if it is a server. Another possibility is that you have to return a broken disk. In both cases, others should not be able to read data stored on the disk.\nAutomatic security updates Every server needs software packages to fulfill its destiny during the lifetime of the system. Ensure that it gets regularly patched and updated by using unattended-upgrades. This is done with the \u0026ldquo;Install security updates automatically\u0026rdquo; option during the installation.\nWhy applying automatic security updates matters: almost daily new weaknesses are detected in software packages. This is no different for Ubuntu servers. Although most administrators rather not update their systems automatically, applying only the security updates is a relatively low-risk action. This is because no new features are introduced, only security flaws are patched. After that is done, a new software update is released to solve the related vulnerability. These updates are often linked to a CVE number (Common Vulnerabilities and Exposures), which provides more information about the vulnerability itself. So don\u0026rsquo;t take risks and apply those automatic security updates.\nMinimal installation The Ubuntu installation has been improved over the last years. It already applies the \u0026ldquo;lean\u0026rdquo; principle. This way it will only install what is really needed. The administrator can still select additional packages or software groups. There is the possibility to add new software groups at the end of the installation process or do it manually later on. Our security tip is to only select the groups and services which are really needed. For server systems, it makes sense to select the SSH server role. This way OpenSSH and the SSH daemon will be installed.\nOnly select what you really need, adding more is still possible later\nWhy a minimal installation matters: Ubuntu has the tendency to enable installed services and start them by default. This means that if you just install a package, it may actually already be running with weak configuration defaults. Security professionals speak often about the footprint or attack surface. This means that the risk of a breach increases if you have more packages you have installed. And not just packages, also the number of services running or (old) users enabled. In other words, each package, service, or user is increasing the chance that your security defenses will be breached.\nHardening steps after installation After you have your system installed, it is time to configure the system. This is also the phase in which your security defenses can easily be weakened. So each time you perform activities on your servers, consider what it does for your overall security level of the server.\nSoftware updates During the installation, there was the option to select automatic security updates. If you already had a system running, you can add this component easily by installing the unattended-upgrades package\napt install unattended-upgrades\nAlthough this package works fairly well out of the box, you might want to check its configuration. This is because only packages from the security repository are updated. Others are skipped. So look for external repositories that may be available to the system and consider adding those packages to the configuration. The configuration file of unattended-upgrades is /etc/apt/apt.conf.d/50unattended-upgrades. For more details regarding the configuration, have a look at our more in-depth article about unattended-upgrades.\nBesides security updates, we suggest to regularly plan a moment to install normal updates. Often they contain improvements to improve system stability. Especially on servers, this is of high importance.\napt update \u0026amp;\u0026amp; apt upgrade\nAccounts Besides the required software packages, systems can only work if it allows users to make use of it. In some cases, we will be giving them access to the server, especially for system administration purposes. Also for web servers, it is common to see that users can access the system directly via SSH and SCP, to allow updating the website of the user.\nRelated risks to user accounts There are a few risks with user accounts on systems, especially on servers. Hardening in this area is therefore required. The first risk is that local users may use local exploits to elevate their permissions and become root. Fortunately, this can only happen if there is actually a known weakness and when the user is skilled enough to run the related malicious script or code. Unfortunately, it is common to see Linux servers running for years, without a single reboot. Even when security updates are automatically installed, an update to the kernel still requires a system restart.\nThe second risk is that old accounts may be lingering on the server too long. Sometimes employees or customers that no longer for the company. When these accounts have a weak password, they might be abused one day. To counter this, set a password policy and delete accounts when they are no longer needed. This last one is a difficult one, as you may need a strict process to control who can access the system and a way to determine the user has no business on the related system anymore.\nConfigure PAM: pwquality PAM is an abbreviation for pluggable authentication module. It extends the existing functionality of the authentication steps, allowing for a very fine-grained configuration. PAM is usually a little bit scary for those who are new to its configuration, as there is not a clear path it follows. Files are including each other and have sometimes cryptic names. Still, don\u0026rsquo;t be scared and test your changes first on a virtual system where you always have root access. Create an additional test user and log in with that, to help with testing.\nThe first step is to install the PAM component \u0026ldquo;pwquality\u0026rdquo;, short for password quality.\napt install libpam-pwquality\nOne of the things we learned earlier is that a longer password is more secure. This is because it usually takes more time to crack by automated tools. So if you want to increase the minimal password length for all users, we have to configure that in the PAM configuration.\nOpen the file /etc/pam.d/common-password in a text editor and search for the line that has the pam_pwquality.so reference in it. By default, it only includes \u0026ldquo;retry=3\u0026rdquo;. This number refers to the retries a user has before PAM will show that the authentication has failed and disallow access.\nAs we want to increase the minimal password length, we add \u0026ldquo;minlen=10\u0026rdquo; just before the retry parameter.\npassword requisite pam_pwquality.so minlen=10 retry=3 Save the file and then switch to your test user in the other terminal. Now use the passwd command and try to use a short password like 123456. If things are properly configured, that should fail.\nNext step is to add complexity rules to pwquality module. This can be done with the parameters: dcredit, ucredit, lcredit, and ocredit. See the man page for their details and again test your changes.\nman pam_pwquality\nBesides this password length and complexity, we can also configure how often a user should change his or her password. This is done via /etc/login.defs with the options PASS_MIN_DAYS and PASS_MAX_DAYS. By default, the maximum days are set to 99999 days and might need to be tuned down. For more sensitive systems this number should be fairly low, like every 30 days.\nFirewall installation and configuration Now that we implemented a few measures, it is time to look at the network services. Even systems that are already filtered by a network-based firewall, might still benefit from a local firewall. There are a few options available when it comes to Linux firewalling, including UFW and iptables.\nFirewall options The best firewall for Ubuntu is the one that you can actually manage. The most common option is iptables . This filtering engine exists for a while and is rock-solid. Its syntax is not that friendly compared with others like pf on BSD. Still, it does the job and gradually you become better at it. UFW or Uncomplicated FireWall is a good option for those that want to apply some simple rules. UFW will take care of generating the required rules for iptables.\nWhy use a firewall on Ubuntu? An important reason to use a firewall is to protect against other systems in the local subnet. Let\u0026rsquo;s say you enabled SSH on all your servers and filtered SSH traffic on your network firewall. Other systems could still use the service internally. In case one system is breached, others might get breached as a result. So one option could be to allow SSH only from predefined systems, like your bastion hosts (a.k.a jump server or stepping stone).\nAnother reason to use iptables or other firewall solutions is to block bad traffic. Sure, it is better if you can do this on a network level, but sometimes only the receiving system can make the decision what traffic is good or bad. Especially when traffic is encrypted, like HTTPS. The receiving server has a better idea on what is going and may decide to start blocking a particular client when it had too many invalid or malicious requests.\nA basic iptables configuration could be looking something like this:\n# Accept all incoming traffic on local interface iptables -A INPUT -I lo -j ACCEPT Next step is to allow traffic to our services:\n# Allow traffic to SSH (to port 2222), SMTP (25), and our web server (80, 443) iptables -A INPUT -p tcp -m tcp -dport 2222 -m state -state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -p tcp -m tcp -dport 25 -m state -state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -p tcp -m tcp -dport 80 -m state -state NEW,ESTABLISHED -j ACCEPT iptables -A INPUT -p tcp -m tcp -dport 443 -m state -state NEW,ESTABLISHED -j ACCEPT Finally, drop all other traffic in the input chain:\niptables -A INPUT -j DROP\nThis simple configuration is just a start. It still allows outgoing traffic, which might need some filtering as well.\nUsing AppArmor security framework Two well-known Linux security frameworks are SELinux and AppArmor. The choice for which framework on systems running Debian and Ubuntu is easy. Only AppArmor is available. This is because only support for this framework is compiled into the Linux kernel.\nIf you really want to harden your Ubuntu systems, then AppArmor is a great addition. This framework defines what running processes can do, or access. Think of it like a prison guard that continuously monitors its prisoners, to ensure they only do activities that are allowed. The allowed activities are stored in policy files, together with the related processes.\nTo determine the current status of AppArmor, use the aa-status command.\nRead more about AppArmor in the Linux security frameworks section\nAppArmor modes: disabled, complain, or enforcing AppArmor can run three different modes. The first of all is disabled. In this mode, AppArmor will simply ignore the policies and not restrict any process. In complain mode, AppArmor will perform monitoring, and only notify when a process is performing an unauthorized action. The most restricted mode is when AppArmor is enforcing. All unauthorized actions are blocked and logged.\nTo learn more about AppArmor, have a look at the AppArmor server guide Next steps for system security Now that you implemented the basic Ubuntu hardening measures, it is time for the next steps. These steps include:\nImplementing sudo OpenSSH security and hardening Using SSH keys instead of passwords Granting Temporary Access to Your Servers (Using Signed SSH Keys) Hardening of compilers and development tools Audit Installed Compilers and Their Packages Check for malware Web server hardening Optimize SSL/TLS for Maximum Security and Speed Detection of malicious traffic Install file integrity monitoring Configure backups Create AppArmor profiles for programs Perform an in-depth Linux security audit We will be adding more links for these subjects with step-by-step instructions.\nUbuntu security tools Security assessment with Lynis If you like to learn what can be improved on your system, use the open source security tool Lynis . This tool is not restricted to Ubuntu. It performs hundreds of individuals tests to detect possible weaknesses of the system. Besides that, it comes with a report that shows suggestions, or room for security improvements on your system.\nMore resources The web has a lot of resources available when it comes to system hardening, including for Ubuntu Linux. Our experience is that there are a lot of low-quality articles, with just some steps to apply and without any reasoning behind it. Avoid those, as they usually have bad examples that can even introduce weak security weaknesses!\nUseful learning resources Ubuntu security notices Do you learn something during following this guide? Great! Become part of the community and share this article on your favorite website or on social media. Got questions or ideas to enhance this guide? Let it know!\n","permalink":"https://linux-audit.com/system-hardening/guides/ubuntu/","tags":["apt","apparmor","compiler","firewall","hardening","security updates","server hardening","system hardening","system security","ubuntu","unattended upgrades"],"title":"Ubuntu system hardening guide for desktops and servers"},{"categories":["Linux","Software","Vulnerabilities"],"contents":"Vulnerabilities happen and are usually fairly quickly fixed. This is also true for Arch Linux. This rolling distribution can be considered to be always up-to-date, as it uses the latest versions of software packages from the upstream. When there is an update, it doesn\u0026rsquo;t take long that it becomes available and can be installed with package manager pacman.\nOne problem that remained was the inability to quickly test if you have any vulnerable packages. After all running pacman -Suy daily works, but that doesn\u0026rsquo;t say much if known issues were found. Till now, with the new arch-audit tool.\nPerforming a vulnerability check The arch-audit tool can be build with the instructions on the website.\nWith the tool being under development, parsing has to be done on the raw text output. With some creative text filtering we can get an output like this:\nUsing arch-audit and some filtering, we can obtain a complete list (with CVE numbers)\nOur security scanner Lynis has support for arch-audit as well. So if you rather don\u0026rsquo;t parse the output and want to perform a security scan daily, then Lynis will do the work for you.\nResources arch-audit (GitHub) arch-audit package ","permalink":"https://linux-audit.com/vulnerabilities/show-vulnerable-packages-on-arch-linux-with-arch-audit/","tags":["arch linux","packages","pacman","software","vulnerabilities"],"title":"Show vulnerable packages on Arch Linux with arch-audit"},{"categories":["Hardening","Linux","System Administration"],"contents":"Feeling overwhelmed with the resources available to secure your Linux system? This security guide will provide you the points where to start.\nWith this Linux security guide, we walk step-by-step through the options, tools, and resources. After reading this article, you will be able to make educated decisions about what Linux security defenses to implement for your systems.\nYou will be introduced to the right tools that help you automate and test your improvements.\nInstead, related articles and resources will be available in the text. The goal is to make this guide into a go-to article for when you need to secure your Linux installation.\nGoals Learn how to select the right Linux distribution Determine which basic security controls are mandatory for each system Select the right security tools for Linux systems How to repeat system hardening on more than one system Know which automation steps can be taken Security steps Select the right Linux distribution The first step for Linux security is selecting a \u0026ldquo;healthy\u0026rdquo; Linux distribution. There are many distributions, each with their own advantages. You may select the distribution on personal preferences, like who maintains it, how commercial it is, or which uses your preferred package manager. We suggest using a distribution that is common, has a clear release schedule, and releases security bulletins.\nLet\u0026rsquo;s have a look at some examples.\nArch Linux The Arch Linux distribution is cutting edge, using the latest software components available. It uses pacman to keep the system up-to-date. As its focus is always to run the latest versions, this distribution is great for research and desktop usage. It may be less suitable for business-critical servers or beginners to Linux. Arch Linux has a plus on security, as it has an extensive wiki with security topics, including security tips for hardening an Arch Linux system.\nCentOS, Fedora, and RHEL These two Linux distributions have a different relation to Red Hat, yet are very similar. For personal use or less critical business activities, these distributions may be a good choice. The package manager on CentOS won\u0026rsquo;t show security related updates. Additional care is needed to keep systems up-to-date to prevent using any vulnerable software.\nFor Fedora, you can choose to show security updates only and keep a system properly patched, even automatic security updates. Fedora is considered to be an experimental Linux distribution by Red Hat, making it more vulnerable to early adopter errors.\nThose with a budget might go for Red Hat Enterprise Linux. RHEL is a good choice if you like the company itself and need stability. As you may expect, Red Hat provides security bulletins and does also contribute to the Linux kernel. It does also do development on security-related products like SELinux. The downside of this distribution is due to stability, most tools are not the latest and the greatest.\nDebian and Ubuntu Debian is well-known for its security, both good and bad. It made some mistakes in the past, like a change that resulted in a weak random number generator. At the same time, it is very keen on having a solid platform to package software and push out security updates when they are available.\nUbuntu is based on Debian and evolved over the years. It is supported by Canonical, and their long-term support (LTS) versions are appreciated by businesses. Both Debian and Ubuntu have the ability to run the AppArmor security framework. They also have people contributing to the Linux kernel like Red Hat does.\nOther distributions There are a lot other Linux distributions and even options like macOS and the BSD family (OpenBSD, NetBSD, FreeBSD, DragonFlyBSD, etc). To learn about more options, we can recommend DistroWatch for a more extensive overview of all the available options.\nChoose security during installation The installation process is the fundament of your system. It will have a great influence on your security posture later on. Typically you want to start safe from the start, as bolting on security defenses on an existing system will consume more time and risk. Doing the right things from the start makes things easier, increasing security and possibly even privacy.\nSeparate file systems When possible separate your file systems, so each of the mount points can be tuned. Most likely you won\u0026rsquo;t need device files on your /tmp partition, which is one of the many things you can do to harden the file system.\nUse the minimal installation option Most Linux distributions have the option to select a minimal installation. Each package has the chance to have a security vulnerability at one time, so only install what is really needed. It also reduces installation time and makes the footprint of the system smaller, resulting in quicker upgrades and better performance.\nEncrypt where possible Some Linux distributions allow you to use LVM with an encrypted file system. This is a great option to protect your data when a disk or complete system is stolen. It also helps when returning disks to a vendor and knowing that vendor will never be able to see any data on the disk.\nUse a strong passphrase to protect the data. If you deploy encryption for multiple systems, consider using a passphrase with an additional string that makes each system unique. For example, something that can be easily looked up by you or your colleagues, yet hard to guess. For example a customer ID, or date of installation. Using the hostname might be less suitable if that is printed on the system itself.\nPrivacy tip: DNS Most home networks are using DHCP to automatically assign an IP address, the default gateway, and the DNS servers. Have a look at what DNS servers are assigned. This might be room for improvement when it comes to privacy. Instead of using the DNS servers of the big companies, consider setting up your own system. DNS is light on resources and can be great to improve privacy for all systems on your network. It even allows you to monitor for infected systems by reviewing the DNS server logs.\nPerforming a security audit System hardening is great, but how do you know you did all that is needed? That is why you perform a technical health scan of your system. We call this process a technical security audit.\nTo perform a security audit you will need the right toolkit. Important is to learn some of the tools and their main purpose. After all, there is no silver bullet in security. Fortunately, Linux provides a lot of security tools . Here is a top 3 for security auditing and vulnerability scanning.\nLynis Nmap OpenVAS Get familiar with at least these three tools. Each of them provides valuable security insights. Also, they will be of great assistance during system hardening and to keep the system secured in the future.\nOngoing audits with Lynis The first tool in alphabetic order is Lynis . This is an open source security scanner which runs on the host itself. It checks the configuration of the system, like a health scan for your body. Any room for improvement is shared in a report. Lynis can also detect vulnerable packages and other weaknesses. A huge benefit of this tool is that it is very light on requirements and resources. In less than 1 minute you can perform a scan. So have this tool installed on your systems.\nPort scanning with Nmap Next in line is Nmap . This versatile port scanner is another battle-tested tool. It helps you to detect open network ports and perform different types of scans. This includes vulnerability scanning and complements the previous tool. Nmap can perform both a check on the local system or do it from the network. It is a powerful tool, which also means it has many options. This can be overwhelming when using it for the first time. Using the tool with some examples from around the web will definitely help.\nVulnerability scanning with OpenVAS Finally, there is OpenVAS . This is an open source vulnerability scanner. It has many tests to perform vulnerability scanning. Like Nmap, it performs its magic by scanning the network. Based on the available systems and services, it will look more specifically at available vulnerabilities. As this tool primarily focuses on vulnerability management, it takes a bit more time to set up than the other two. You will need a dedicated system with enough resources and have it update its database first.\nDon\u0026rsquo;t run once: schedule These three tools combined provide a good basis to detect weaknesses and continuously perform security checkups of your systems. Schedule Lynis to run daily on the machine. Have it email the results, or store the output on a central system. Use a daily difference check to show what has been changed. Do the same for nmap, to ensure no unwanted ports are opened unexpectedly. If you are using a mixed environment, nmap has also a way to show Linux systems only.\nSystem hardening for Linux systems Now it is time to start hardening the system. System hardening is the process of adding new defenses and removing weak spots in existing defenses. We remove these weak spots by minimization, detection of vulnerabilities, and adjusting weak configuration defaults.\nIf you are new to Linux security, then read the article How to secure a Linux system. It provides a good introduction to these principles.\nPrinciple of minimization Every piece of unneeded ballast on the system should be removed (as much possible). This includes users, processes, files, directories, empty log files, etc. Sure, don\u0026rsquo;t overdo it, as it may result in a broken system. Often you can remove a lot without any impact to the system.\nFor example when atd is running, but you don\u0026rsquo;t use that scheduler, then simply remove it. You can find the related package a file belongs to, then remove that package. The same applies to unused user accounts. The article Unused Linux users: delete or keep them? provides background on user management.\nUpdate software packages The easiest way to kill a lot of vulnerabilities, or software weaknesses, is by installing the related software updates. An update is a patched version of a software package that has a known issue. This could be a critical bug causing data corruption, a new feature to make life easier, or solving a security weakness.\nWe speak about updates when it slightly changes the behavior of a program. Often these are minor releases (1.2.3 to 1.2.4, or 1.2.8 to 1.3.0). An upgrade is usually bigger, like moving to a new major version, or a new operating system version (Ubuntu 16.04 to 18.04). Both upgrades and updates help with getting better software and make system management easier. While there is always the chance of breaking some functionality or process, this should be no excuse to stop doing proper patch management. Use clusters, snapshots, and backups to your advantage, to allow regular software updates to be installed.\nWeak configuration details Most software is created to work as simple as possible. That often conflicts with the principle of secure by default. So for every package we install, we should carefully look at the impact it has on the system. Ask yourself the following questions:\nDoes it spawn a new process? Does it listen on the network interfaces by default? Does it require authentication in any way? For each of these questions, we should find the answers and make an educated decision on what that might mean for the system.\nNetwork traffic filtering Linux allows filtering network traffic with the netfilter functionality. You may be familiar with the tools that use this kernel feature:\niptables nftables firewalld bpfilter Kernel support for netfilter is usually already enabled by the Linux distribution, so no recompilation is needed.\nEven with a network-based firewall enabled, you may benefit from adding a local firewall. It will decrease the chance that a successful attacker may jump between hosts, as traffic streams to (and from) the system can be regulated. It does also allow tools like fail2ban to create a blacklist for that specific server, based on locally gathered intelligence.\nSecuring authentication on Linux Linux systems usually have the PAM framework available. The abbreviation stands for Pluggable Authentication Module. It provides a stackable set of authentication modules. This stack then determines who can access the system and any specific conditions that might apply to the session. Not only does it filter out the authorized users, it can set shell specific settings and check for password strength.\nImprove your passwords You can use very weak passwords on Linux systems by default. That might be fine for your personal system, but less suitable for corporate systems. In such case have a look at pam_cracklib or the newer pam_pwquality. With these modules, the minimum password length can be enforced, or the variety in special characters that should be available. If you are used to two-factor authentication, consider using that for your most sensitive systems.\nSSH hardening Most systems use SSH to be managed. So it shouldn\u0026rsquo;t be a surprise that this access point needs some attention as well. If you run the OpenSSH daemon, perform SSH configuration hardening before deploying the system. If it is already running, then consider who is connecting to it, and restrict access using options like AllowUsers.\nDue to the importance of this subject, we have some other related articles to SSH hardening. Determine what would make sense in your situation.\nUsing SSH keys instead of passwords Distributing SSH keys: using ssh-copy-id, manually or automated Using Ed25519 for OpenSSH keys (instead of DSA/RSA/ECDSA) Granting Temporary Access to Your Servers (Using Signed SSH Keys) Audit file access and events One of the least used options on Linux is still the Linux Audit Framework and the related audit daemon with kernel integration developed by Red Hat. Although setting it up takes a little bit of time (and testing), it is very powerful to detect file changes and reporting the usage of specific system calls. Great for intrusion detection or even as a tool to troubleshoot.\nAudit commands by users Your security policy might be stating that all commands need to be logged. You can use the Linux audit system to capture commands executed by the root user, or any other user for that matter. But there are also other ways to capture some events on the system. In such case, you might want to configure Snoopy, a fairly easy way to capture that information.\nContainers Although container technology isn\u0026rsquo;t new, the implementation on Linux is fairly young. Containers help to encapsulate a service into its own bubble. This has a few advantages, like having a better picture of what kind of traffic needs to flow between applications. Containers are controlled using control groups and namespaces. This keeps them under control and limits the maximum resources they can use. Within the container, you can only see your own processes and user. From the outside (on the host itself), things look normal. In the end, each container is still a normal process.\nWhile the container technology is taking on, there is still a lot of development going on. Most of it is in the supporting tooling, to make management easier. If you truly want to understand the capabilities of containers, I urge you to have a look under the hood and understand the techniques used. A few of them are:\ncgroups namespaces seccomp port filtering read-only file system Protect databases When running a database like MySQL or PostgreSQL engine on your system, additional care should be taken for the system. As usually precious data is being stored, the first step is to check if your backup strategy is properly implemented. A normal file backup might be insufficient for proper database backups. Instead, create a database export with tools like mysqldump or pg_dump.\nThe next layer of database security includes setting up proper authentication. For example, ensure that the local \u0026lsquo;root\u0026rsquo; user needs to authenticate. This is a common finding of Lynis, where simply no password is set. Use the strongest hashing algorithms as possible for authentication options, also if you have a custom web application. In such case use hashing together with salting. Your programming language should have the appropriate libraries to achieve that.\nMySQL and MariaDB my.conf (configuration) mysql_secure_installation (hardening tool) PostgreSQL pg_hba.conf (configuration) Hardening a web server Setting up a web server is fairly easy. Protecting it properly against evil bots and attackers is a whole different story. A few things that every web server contain now is being HTTPS-only and restrict access to sensitive files. When possible a web application firewall (WAF) is a great addition to block SQL injections and other common attacks.\nAs setting up a web server in a secure way is an article in itself, here are some pointers to improve your web server.\nSecuring nginx configurations: implementing OCSP stapling Configure HSTS (HTTP Strict Transport Security) for Apache/Nginx Optimize SSL/TLS for Maximum Security and Speed Hardening WordPress About 25% of the web is powered by WordPress. That is a nice achievement, yet means the software is a target as well. Keep your installation up-to-date and especially update plugins on a regular basis. Remove plugins that are badly maintained and replace them with others. Yes, your website functionality might need to be changed a bit, but that is always better than have to deal with a breach.\nMore resources Understanding Linux privilege System hardening provides additional layers of defense against malicious attempts to breach the system. To properly increase our Linux security defenses, we need to understand how attackers work, and in particular how Linux privilege escalation works. Like Sun Tzu said, you have to understand both your enemy and yourself. Only then you don\u0026rsquo;t have to fear the battle.\nLinux security best practices and hardening guides It is common to find \u0026ldquo;best practices\u0026rdquo; in the world of information security. These are commonly configuration settings and practices that are considered to be safe. For most companies and environments these are suitable. As always, you have to confirm if this also applies to your systems. Hardening guides are usually filled with these best practices.\nSecurity conferences and video presentations Although there are many conferences, there aren\u0026rsquo;t many that just focus on Linux security. If you are interested in video materials and presentations, check out the following resources.\nLinux security summit O\u0026rsquo;Reilly security conference If you like this article, become part of the community and share it with your network. Got feedback? Use the comments at the bottom.\nWhat additional defenses are you using and did we skip in this article?\n","permalink":"https://linux-audit.com/linux-security-guide-extended-version/","tags":["bpf","cgroups","dnf","guide","hardening","linux security","lynis","namespaces","nmap","openvas","system hardening"],"title":"Linux security guide: the extended version"},{"categories":["Software","System Administration"],"contents":"Sometimes you want to know the related package of a file, before installation, or when it is already there. This is of great help during system hardening or general system cleanups. In this article we have a look at several ways to determine the relationships between files and the package they belong to. We have gathered this information for multiple Linux distributions.\nMost options used in this article have also a long format option. When using options in shell scripts, it can be a good idea to use these full names, as they are typically more clear on their purpose.\nArch Linux Related package (installed) pacman -Qo binaryname\nShows the related package and the binary path for installed packages.\nShow files provided by the package Another option is using the pkgfile command.\npkgfile -l packagename\nNote: most likely this tool needs to be installed first: pacman -S pkgfile \u0026amp;\u0026amp; pkgfile -u\nCentOS, Fedora, RHEL Show files for RPM packages The rpm command can be used to query information about RPM packages, including those that are installed or RPM files.\nrpm -qlp /path/to/file.rpm\nThese options translate to the following actions below.\nOption Long format Related action -l --list List files in package -p --package Define the related package to use for the action -q --query Perform a query action Show files for packages in available repositories If you use dnf, then you can query files from the packages that are in your repositories. The package itself does not have to be installed.\ndnf repoquery -q -l packagename\nUse the -q option with dnf to show only the relevant output.\nShow files for an installed package To show what files are provided by an installed package, use the rpm command.\nrpm -ql package\nIf you have the file name, you can turn this around and find the related package.\nrpm -qf /bin/ps\nThe output will provide the package and its version.\nTo just see the package name, use the \u0026ndash;queryformat option.\nrpm -qf /bin/ps --queryformat '%{NAME}'\nWith yum you can do a similar request to see the related package.\nyum whatprovides /bin/ps\nAnd with DNF there is the provides argument.\ndnf provides /bin/ps\nThis will give you possibly multiple hits, as a file can be part of packages from different repositories.\nThis data is less easy to parse due to the different types of lines.\nDebian and Ubuntu Discover related package If you want to find the related package of a binary (or file) on Debian or Ubuntu, we first have to know the full path using the which command.\n# which ls /usr/bin/ls Now that we know the location of the binary, then use the dpkg command with the --search option to discover where it is stored.\n# dpkg --search /usr/bin/ls dpkg-query: no path found matching pattern /usr/bin/ls This might be unexpected, as the binary definitely should be coming from some package.\nThe reason for error dpkg-query: no path found matching pattern is that on newer systems the directory /bin might be symlinked to /usr/bin. The package might be install a binary in /bin, while the which command finds its first result in /usr/bin.\nTo resolve this, we can combine both commands and tell which to return all entries using -a.\n# dpkg --search $(which -a ls) dpkg-query: no path found matching pattern /usr/bin/ls coreutils: /bin/ls Now we know that the ls command comes from the package coreutils.\nShow files installed by package If you already know the package name, you can quickly look up the files that are installed by a Debian package.\ndpkg -L package\nLet\u0026rsquo;s do the same for the at package and see what it exactly installs (and where).\nGentoo The first option is using equery, which is part of the package app-portage/gentoolkit.\nequery files \u0026lt;installed package\u0026gt;\nThe package itself should be installed.\nNext alternative is qlist, which is part of app-portage/portage-utils\nqlist name-of-installed-package\nOpenSUSE Systems running the distributions from SuSE can use the zypper tool to find the link between a file and a package.\nShow related package zypper what-provides /bin/ps\nGot more useful commands to share? Let it know!\n","permalink":"https://linux-audit.com/software/package-manager/determine-file-and-related-package/","tags":["awk","debian","dnf","dpkg","gentoo","linux","package manager","packages","rpm","ubuntu","yum","zypper"],"title":"Discover to which package a file belongs to"},{"categories":["System Administration","Tools"],"contents":"The grep command is one of the oldest tools for Linux and other platforms. Actually, it is older than Linux itself. It was written by Ken Thompson more than 45 years ago! The name grep stands for \u0026ldquo;globally regular expression print\u0026rdquo;. This name comes from its predecessor ed and the specific mode in which you would globally search, using a regular expression, and print the output. The related command was \u0026ldquo;g/re/p\u0026rdquo;. For more history, have a look at the Wikipedia entry. Otherwise, let\u0026rsquo;s dive into the tool and get to know some practical grep examples for daily usage.\nIntroduction One of the reasons to create this blog post is that there are a lot of examples available for the grep command. But with all information scattered, most people don\u0026rsquo;t take the time to really learn the most basic commands. We want to leverage the full potential of the grep command, as it can be used in many work-related or personal related activities. It is common to use it for checking configuration files and searching through log files.\nWhy learn the grep command and regular expressions? As with every tool, it is often easy to start using it, but hard to really master it. The man page is very extensive, so is the online help documentation. Although these sources are a great reference, we will be showing the grep command by example. And we will include specific use-cases which are common for system administrators and security professionals. Especially if you have to deal often with data, investing some time in doing things efficiently will pay off.\nBefore you continue If you are using grep on another platform than Linux, you may not have the GNU version of grep. Some things in this guide may not be working, or need specific tailoring. You can easily find out what version you have with grep --version.\nNeed a particular job to be done with the grep command and can\u0026rsquo;t get it to work? Use the comments and share what you have tried. Let\u0026rsquo;s start with the basics and become a \u0026lsquo;grep master\u0026rsquo;.\nBasic usage examples of grep Use grep for simple actions The grep utility does not need much to starts doing its work. The syntax of grep consists of four parts.\ngrep command optional: option(s) string to search file, files, or path to be searched The options that grep uses typically have a long and short format. The long format has two dashes, followed by a word or words. Use the long format when using them in scripts, so that it becomes obvious what the grep command is doing. Use the short notation in your daily tasks and on the command line, to save on typing and speed up your work.\nIf you would like to find the root user in your /etc/passwd file, just tell it to search for \u0026lsquo;root\u0026rsquo; and the file name itself. In this case, no option is needed.\ngrep root /etc/passwd\nUsing colored grep output If the command above did not show colored output on your system, you might want to enable that. It can be done with --color auto. As this would mean you have to type it in each time, using an alias would save you from a lot of typing.\nalias grep='grep --color=auto'\nYou can add this alias to your .bash_aliases or .bashrc file if you are using the bash shell. Otherwise, add it to the respective profile file. These files can be found in your home directory.\nIgnore case sensitivity Now that we have performed a basic grep command, we can start to change its behavior. Often we already know the word or words we are looking for. What we don\u0026rsquo;t always know is if one or more occurrences of the word are using capitals. By default, the grep command will be case-sensitive. So only the right match will be displayed. We can tell grep to ignore case-sensitive searches with the -ioption.\ngrep -i root /etc/passwd\nShow line numbers Depending on your search, you may have many occurrences of the text you were searching for. Use the -n option to have grep show the related line numbers.\ngrep -n root /etc/passwd\nExcluding words To exclude particular words or lines, use the -invert-match option. Use grep -v as a shorter alternative. Exclude multiple words with grep by adding -E and use a pipe (|) to define the specific words. Optionally make it case insensitive with the -i as listed above.\ngrep -i -v -E 'banana|monkey' zoo.txt\nMatch counting It may be useful to know the number of occurrences of your specified word. This count is displayed when using grep -c or grep -c -v to show the number of non-matching lines.\ngrep -c monkey zoo.txt\nRecursive search through directories and files To search in one directory, there are the -r and -R options to achieve this. Depending on the target and the existence of symlinks, you might want to use the first one if you do not want to follow them. Use the capitalized option, grep -R, if you want to include any possible symlinked file to be searched as well. This may take much longer and could result in other file systems to be searched as well.\ngrep -r password /etc\nTip: if you don\u0026rsquo;t want the filenames in the output, add the -h option.\nShow matching files only Sometimes you just want to see the files that match a particular text string. There is the grep -l command to do achieve this.\ngrep -l -R password /etc\nTo show all files that do not match your target, use the capitalized version: grep -L.\nUsing regular expressions The grep utility is a powerful tool and can use regular expressions. Regular expressions can be considered \u0026rsquo;logic rules\u0026rsquo; for matching text strings. Think of something like \u0026ldquo;I know the word should be starting with the letter \u0026lsquo;a\u0026rsquo;, but after that everything is fine\u0026rdquo;. By using a regular expression we can express this in short notation (e.g. \u0026quot;a.*\u0026quot;).\nMatch specific words only You may be searching for a very short, yet specific word. In this case, grep will return way too many results. By using more specific statements we can limit the output.\ngrep \u0026quot;\\bbin\\b\u0026quot; /etc/passwd\nThe \\b tells grep to use word boundaries.\nAlthough you could use spaces to search for a full word, that often won\u0026rsquo;t give you the right result. It will return some hits, while it might be missing a few as well. For example, any occurrences at the begin or end of the file. There will also be no match if any special characters are followed by it, or even a simple character like a comma.\nTip: use the -w(--word-regexp) option to achieve the same as this regular expression above, as it is easier to remember.\nFind lines starting with a specific string With the carrot symbol (^) we can activate a regular expression that defines that the line should start with a specific piece of text.\ngrep \u0026quot;^systemd\u0026quot; /etc/passwd\nFind lines ending with a specific string Like the carrot symbol, we can use the dollar sign ($) to mark the end. Only lines that match that, will be returned. A great way to find all accounts that have a particular shell configured.\ngrep \u0026quot;bin/bash$\u0026quot; /etc/passwd\nSearch for multiple words Sometimes you want to match multiple words. By using parentheses you can tell grep to search for one word, or the other. Each possible match is split by a pipe sign.\ngrep -E \u0026quot;^(backup|root|syslog)\u0026quot; /etc/passwd\nMatching multiple words\nNote: use the -E option to enable extended regular expressions. Without it, the command won\u0026rsquo;t give any results.\nCombining grep with other tools Exit code Using grep in your shell scripts can be very useful. For example, you can use it to determine if a particular file has the right configuration setting and then perform an action based on that. Another one is to see if a particular user exists in your /etc/passwd file.\ngrep -q michael /etc/passwd\nGrep will not display anything, but end with an exit code. This exit code will be stored in a special variable with the name $?. If you want to see it on the command line, use it with echo.\necho $?\nExit codes:\n0 = match found 1 = no match found 2 = error Example syntax to use grep in your shell script:\nif $(grep -q michael /etc/passwd); then echo \u0026quot;Michael is in passwd file\u0026quot;; else echo \u0026quot;Michael is not in passwd file\u0026quot;; fi\nUsing pipes The grep command is a great utility to use in combination and filter the output of other commands. This way the screen only shows that data you are interested in. To achieve this we use the pipe sign (|) to tell the shell to send any output to the next command in line.\nIt is common to apply multiple grep commands by piping them together. When using big data files, try to limit the number of pipes to increase performance. You may also want to look for alternative solutions when you are repeating them often.\nExample: Search in dmesg output The dmesg command gives a lot of lines as output. If we are just interested in information regarding our storage, we can easily do by searching for \u0026ldquo;sd\u0026rdquo;.\ndmesg | grep sd\nIf we just would like to find AppArmor related events, it would make sense to ignore case due to the capitals in the name. By smart combining the right tools, we can form a powerful data filter.\ndmesg | grep -i apparmor\nAdvanced tips Improve search speed: fixed strings Typically you may be using already a specific word that you want to be matched. When searching through big files, grep may take a while to complete its task. By using the -F (fixed strings) option this can be dramatically improved. The only downside is that regular expressions can not be used.\nSearching inside compressed data (avoid using gunzip!) Need to search inside compressed files? Use the zgrep command. It has the same syntax and it knows how to deal with compressed data.\nConclusion The grep command is a very powerful tool and easy to work with. To truly master it, one should be learning more about regular expressions. It makes searching and finding the right data much easier. Knowledge about regular expressions will also come in handy for other tools, like sed and awk. If you really want to learn how to use the grep command, use it daily and create your own list of commands you often use.\nDo you have a great one-liner that you often use with grep? Share it and we see if it can be included!\n","permalink":"https://linux-audit.com/grep-commands-and-common-examples-for-daily-use/","tags":["grep","how-to","linux","system administration","tutorial"],"title":"How to use grep (with examples)"},{"categories":["Pentesting"],"contents":"The information security field is filled with all kind of tests and assessments. One of them is the penetration test, also abbreviated to pentest or pen test. Last years, many security consultancy firms offer this test as part of their security services. So what is it really and when should you undergo a penetration test? Continue reading!\nWhat is a pentest? The short question to what a penetration is: a hack attack on your environment, executed by professionals, with approval and a written understanding (or contract). The attacking party has your permission to perform possibly harmful actions to your network, data, and people. Although most of the actions might be technical, it could also include non-technical steps. One of them might be tricking your staff to provide access or sensitive information, which we call social engineering.\nScope and jail Within the penetration test there is a clear scope on what is included, and what not. If you only want your external web servers to be targeted, then the firm performing the penetration test is limited in what it can do, or is allowed to do. Another key component is the permission you provide. There should be a written statement that you ordered the security firm to execute possibly harmful actions. And if you do things correctly, you clearly state what is not allowed, like peeking in the mailbox of the CEO. For some assignments, the pentesters even get a \u0026ldquo;Get Out of Jail Free\u0026rdquo; card, which they can show to security guards or IT staff, in case they get caught during the assignment.\nBesides scope, there might also be restrictions in the time that the assignment can happen. This could be the specific moment of when the actions happen or the total time of the assignment (e.g. one week). And to add another level, penetration tests might be \u0026ldquo;white box\u0026rdquo; or \u0026ldquo;black box\u0026rdquo;. This means that any information is shared upfront (white box), or none at all (black box).\nLet\u0026rsquo;s use an example: company Acme has a website running on Red Hat Linux. The website accepts online payments, including credit cards. The company needs to be PCI DSS compliant and as part of the standard is required to have a penetration test performed. They ask the fictional firm RTP (Red Team Pentesters) to perform a penetration test on their website and Linux infrastructure.\nSteps involved Every well-executed pentest is performed by finishing the activities ordered in several phases. While each company has their own approach, there is a generic way of working. These steps are:\nReconnaissance Enumerate Exploit Document Let\u0026rsquo;s go through these steps quickly to learn why these phases are important.\nStep 1. Reconnaissance When the penetration test is started, the \u0026ldquo;attackers\u0026rdquo; start with collecting information about the target. They want to gather as much information as possible, before launching any other actions. This first phase is called reconnaissance. We might learn from job postings on the web that Linux is used, together with Apache and MySQL.\nStep 2. Enumeration After the first round of information gathering, more in-depth tests are being performed. This is called enumeration and can be considered a technical term for interrogating the system. For example, during reconnaissance we learned there is a website and who owns the domain name. We might now do an actual attempt to learn what software components are used. This can be done by looking at the HTML code of the website or request non-existing pages to see if there are information leaks. During the penetration test our fictional firm RTP might learn that WordPress 4.5.1 is being used, together with several outdated plugins.\nStep 3. Exploitation With all the information we gathered we can now try actual attempts against our target systems. Program code is executed to test if it triggers an expected or unexpected result. This might be leaking more information, or finding actual vulnerabilities. A vulnerability is usually a program error or weakness the software itself. The actual code being executed is called an exploit, hence the exploitation phase.\nIn this case, the pentesters of RTP find that they could use a successful exploit against the WordPress code. They gained access to the system and in steps retrieved more details step by step. Finally\nStep 4. Documentation and reporting Any successful achievements will have to be documented, like retrieving the password of the system administrator. It includes the steps taken on how the \u0026ldquo;price\u0026rdquo; was obtained, together with advice. Such advice could be related to protect against the used attack, or how to solve a discovered vulnerability.\nThe challenges with pentesting With all the scoping and types and possibilities, pentesting looks like a customized project. That is actually the right way to approach it. It should have a clear begin and end, and define what outcome would make it a successful assessment. This is where things get difficult.\nSkills matter When a security company succeeds and is able to break in, you can say it was the right choice to do it. On the other hand, when the security company is not able to achieve a certain outcome (like obtaining administrator credentials), does that mean things are really secure? Maybe the skills of the related people are lacking, resulting in weaker capabilities to obtain the precious crown jewels. In other words, the pentest is as good as the people involved. Most systems can be breached if you have skilled personnel doing the pentest and give it enough time. With this thought in mind, all parties involved should be honest about the possible outcomes, the available time, and related costs.\nVulnerability assessments != pentest Some penetration tests executed are actually vulnerability assessments. Instead of a well-executed penetration test that is based on a good understanding of the goals, a simple list of vulnerabilities is shared with the client. Although it may uncover flaws in the environment of the requester, it might lack the depth which a penetration test can provide.\nToo early One of the biggest mistakes that may happen is related to the title of this post: when should you do a pentest? Unfortunately too many companies do it too early. Sometimes because they are required by others, for compliance reasons, or being sold the magic powers by security firms. Consider that you want to test if your house is safe against burglars. Would it make sense to have a security assessment if you already knew there was not a lock on the front door, and no alarm system was present at all? Probably not. This is what often happens when the idea of a pentest is sold to the IT manager or owner of a company.\nCorrect order: Audit, Vulnerability Scan, Penetration Test If you consider the phases a penetration tester performs during an assessment, we can apply the same steps. Each step gets a specialized security scan of its own.\nStep 1: Reconnaissance = Audit The first step was reconnaissance, or gathering basic details. This is similar to perform an audit. This may be a generic IT audit, looking at documentation in particular. This includes processes and the implementation of them in the daily IT practices of the company. On a technical level, we can perform a system audit. This gives a better understanding of the quality of all technical measures implemented and any room for improvement. It may include system hardening, intrusion detection capabilities, and dealing with automated security patching. Audits may also help with determining compliance with internal baselines and external standards like ISO27001/ISO27002, PCI DSS, and HIPAA.\nSome audit tools can also help with information collecting and storage similar to a configuration management database (CMDB). The CMDB provides insights and details on the systems, their configuration, and ownership.\nRelated Linux software during this audit phase includes:\nLynis nmap Step 2: Enumeration = Vulnerability Management When the right procedures are in place and systems are properly hardened, the next level is digging in the details of our software. With software being a regular reason for systems being compromised, we can perform vulnerability scans. We look specifically on the software being used and see what possible exploits could be used against it.\nRelated software for Linux systems include:\nnmap (with plugins) OpenVAS Step 3: Exploitation = Pentesting Only when we have the right procedures in place and we did a deep technical scan of our environment, pentesting will be useful. Although security is an ongoing process, at some stage we should be confident enough that we did the right things. Now it is time for others to perform simulated (and real) attacks against our environment.\nRelated software during the exploitation phase includes:\nMetasploit Kali Linux (multiple tools) Step 4: Documentation = Reports The last step is about doing something with the data that some of the mentioned tools above provided. This could include live dashboards, PDF reports, or plain textual output. Especially program output can be powerful when collecting it centrally and parsing it into formats that your colleagues can use to plan the next cycle of improvement.\nConclusion Penetration testing is a powerful measure to test your security defenses. Its value really shows when security is already part of the company culture, with the basics properly implemented. Then the pentest may uncover flaws that were missed before and encourages continuous improvement.\nPentests needs to be properly scoped, together with the right amount of time. This way both the security firm and tested company have a better assurance that the right things were tested and can withhold serious attacks.\nFor the Linux platform there are many tools to help with auditing, scanning for vulnerabilities, or even distributions that guide the pentest. Some are mentioned above. Got some great tools you use? Let it know!\n","permalink":"https://linux-audit.com/when-should-you-do-a-penetration-test/","tags":["penetration testing","pentest"],"title":"When should you do a penetration test?"},{"categories":["File Systems"],"contents":"When looking in /proc you will discover a lot of files and directories. Many of them are just numbers, which represent the information about a particular process ID (PID). By default, Linux systems are deployed to allow all local users to see this all information. This includes process information from other users. This could include sensitive details that you may not want to share with other users. By applying some file system configuration tweaks, we can change this behavior and improve the security of the system.\nHiding processes for other users Since Linux kernel 3.3 there are two new mount options for the Proc pseudo-filesystem. The first one is hidepid, to hide process IDs. The second one is gid, to allow some users to see information, even though it is blocked with the previous hidepid.\nNormal users can see all process IDs\nIn this example, we can see that a non-privileged user can see all process identifiers (PIDs). If you would like to see what process is involved, simply use the cat command.\ncat /proc/[ID]/cmdline\nThis command will display the related binary that called to start the related process. It will also include the parameters that were provided. As you may have expected, this is also how the ps command is able to show this information.\nHardening /proc with hidepid To dynamically test the impact of the hidepid mount option, you can remount the /proc partition. This needs to be done as the root user or by using sudo.\nmount -o remount,rw,hidepid=2 /proc\nWhen the same non-privileged user tries to display the information now, only process IDs of his own user will show up.\nThe /proc mount is now hardened with hidepid=2 option\nAlso using utilities like ps and top will now only show your own processes. A great way to prevent sharing a lot of information about the system and the processes running on it.\nIf you like to make the change permanent, change your /etc/fstab file and reboot the system.\nproc /proc proc defaults,hidepid=2 0 0 Values of hidepid By default, the hidepid option has the value zero (0). This means that every user can see all data. When setting it to 1, the directories entries in /proc will remain visible, but not accessible. With value 2 they are hidden altogether. This last option will work perfectly for most systems.\nGiving some users permission to see all processes You may want to use the hidepid option, but have software which depends on seeing all the processes. In that case, you can add the gid mount option. This tells the kernel that users in that group (and root) can still see the information. The group itself is referenced by its group number. For example, you could create a group monitoring, and then allow this group to see all processes.\ngroupadd -g 1500 monitoring\nDid you learn something from this article? Great! Share it with others, like your favorite website or social media. Got some additional tips? Let it know!\n","permalink":"https://linux-audit.com/linux-system-hardening-adding-hidepid-to-proc/","tags":["file system","hardening","mount","system hardening","system security"],"title":"Linux system hardening: adding hidepid to /proc mount point"},{"categories":["System Administration"],"contents":"Software updates and package management is easy with systems based on Debian or Ubuntu. Just apt-get update (or apt update) and run an upgrade. But sometimes you may encounter the following situation: a KEYEXPIRED message.\nKEYEXPIRED message # apt-get update \u0026amp;\u0026amp; apt-get upgrade Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB] Hit:2 http://nl.archive.ubuntu.com/ubuntu xenial InRelease Get:3 http://nl.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB] Hit:4 http://nl.archive.ubuntu.com/ubuntu xenial-backports InRelease Hit:5 https://packages.cisofy.com/community/lynis/deb stable InRelease Get:6 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [373 kB] Ign:7 http://nginx.org/packages/mainline/ubuntu xenial InRelease Get:8 http://nginx.org/packages/mainline/ubuntu xenial Release [2,309 B] Get:9 http://nginx.org/packages/mainline/ubuntu xenial Release.gpg [287 B] Get:10 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [368 kB] Get:11 http://nl.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [319 kB] Get:12 http://nl.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [316 kB] Err:9 http://nginx.org/packages/mainline/ubuntu xenial Release.gpg The following signatures were invalid: KEYEXPIRED 1471427554 Fetched 1,566 kB in 0s (2,003 kB/s) Reading package lists... Done W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://nginx.org/packages/mainline/ubuntu xenial Release: The following signatures were invalid: KEYEXPIRED 1471427554 W: Failed to fetch http://nginx.org/packages/mainline/ubuntu/dists/xenial/Release.gpg The following signatures were invalid: **KEYEXPIRED** 1471427554 W: Some index files failed to download. They have been ignored, or old ones used instead. Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: apparmor libapparmor-perl libapparmor1 python3-distupgrade python3-software-properties software-properties-common ubuntu-release-upgrader-core 7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/672 kB of archives. After this operation, 5,120 B of additional disk space will be used. Do you want to continue? [Y/n] y The KEYEXPIRED shows that validation failed on the related repository signature. This is a good thing, to warn us that we should be checking the repository. With an expired key, the solution is simple: we need to download an updated key. Apparently it is for the nginx repository.\nStep 1: Run apt-key Using the apt-key utility we can display all the known keys.\napt-key list\nIn our case, we see the nginx key is expired a few days ago:\npub 2048R/7BD9BF62 2011-08-19 [expired: 2016-08-17]\nuid nginx signing key signing-key@nginx.com\nTwo items are highlighted in this example. The first one is the short version of the key. The second one is showing that the key is expired (including the date). This key was valid for almost 5 years.\nTo quickly find the expired keys, search for \u0026ldquo;expired:\u0026rdquo;:\napt-key list | grep \u0026quot;expired:\u0026quot;\nStep 2: Update the key We can now use the key gathered in step 1 to update it:\napt-key adv --keyserver keys.gnupg.net --recv-keys [KEY]\nThe output might look like this:\nThe key is renewed, after choosing the right one (otherwise no change is made)\nOn purpose we selected an incorrect key, which was also related to nginx:\n/etc/apt/trusted.gpg.d/nginx-development.gpg pub 1024R/C300EE8C 2010-07-21\nuid Launchpad Stable\nAs you can see in the output above, nothing happens when you select the wrong key.\nStep 3: Update After renewing the expired key you can run apt update again and install any available upgrades.\napt update \u0026amp;\u0026amp; apt upgrade\nHappy upgrading!\n","permalink":"https://linux-audit.com/how-to-solve-an-expired-key-keyexpired-with-apt/","tags":["apt","debian","how-to","linux","software management","ubuntu"],"title":"How to solve an expired key (KEYEXPIRED) with apt"},{"categories":["Hardening"],"contents":"When it comes to Linux system hardening there is a lot to do. From the almost book-like CIS benchmarks to following best practices found all over the web. Recently someone new to the field of information security asked me a simple, yet important questions: how much system hardening should you be doing? When is it enough? Since there was no easy answer, I have written down my thoughts to help others in the future.\nTime and Effort To understand the value of system hardening, one should know that it is a matter of putting time and effort into it. Time is simple: you have to first recognize that there is something to improve and decide to invest time to do research. Then it is about finding the right resources to help during the system analysis and system hardening process. You should first know what you can improve and test if things are actually related to your environment. Then more time goes into making a decision on what defensive measures to use, test them, and finally implement them. And you might also do some monitoring afterward, or even solve issues caused by your actions. So there is a lot of time involved for sure.\nWhen it comes to effort, that one depends on your skill set and tooling used. Regular readers of this blog might already know that you don\u0026rsquo;t have to check every setting by hand. Tools like Lynis provide a quick and extensive method to discover possible security gaps. What also helps is your own skill set to quicker making an estimation on what would, or would not, work in your environment. After all, security is a matter of security appetite. You decide how much risk you want to take.\nThe value of starting Every beginning is difficult. With system hardening that is not different at all.Getting started somewhere is the key. For that reason, the Enterprise offering of Lynis provides a section called \u0026ldquo;Improvement Plan\u0026rdquo;. It searches through all the findings and provides you with some quick wins. Start with the simple things and make quick decisions. Nike\u0026rsquo;s slogan \u0026ldquo;Just Do It\u0026rdquo; is applicable here. Making small steps, like removing the greeting banner on Postfix. Why would you tell everyone that you are using Postfix? Read, understand, and implement that new banner that says \u0026ldquo;mydomain.com ESMTP\u0026rdquo;. You are now started with system hardening and slightly decreased the chance that the average Joe can discover what software components you are running.\nMarathon versus Sprint When we talk about information security, we should consider that we are running the marathon. There is no use in implementing security quick if you can\u0026rsquo;t keep up. It is better to craft a culture of continuous improvement. Become a little bit better (and more secure) every day. So considering that way of thinking, we can answer the question at the beginning of the article with: \u0026ldquo;never\u0026rdquo;. There is obviously some nuance to it.\nThe level of security should be high enough. If you don\u0026rsquo;t have enough of it, you will later discover the consequences (a possible break-in, data leaks, damage to company brand). Too much security will actually harm the business as well. When simple tasks become too complicated through all level of implemented measures, the business will slowly grind to a halt.\nKnowing when to stop Finding the right level of security does not just depend on your organization. If you are a bank, you know that trust is everything. So that means you will have to implement those measures that display trust and to some extent even can guarantee it. If you are any other business, then your security posture might be completely different compared to your competitors. That is fine, as long as you know your risks, threats, and vulnerabilities. Not just on business or financial risk, but also technical risks. Getting a clear picture will help you focus on the right things. That system that deals with credit card payments might need a lot more security measures than your developer system. Well, unless your intellectual property is stored on that machine, and contains even a higher value.\nVulnerability Management One of the things I dislike from the information security field is vulnerability management. It has even become a business model, fed by fear, uncertainty, and doubt (FUD). Instead of helping people with the next step, many solutions work on presenting all the weaknesses. Sure, you have to measure something. The focus should be on the positive (implementing measures), not on the negative outcome. You would be surprised to find how many quick wins are not implemented, while these same companies invest lots of money in all kind of security hardware and software. What we need are tools that educate us, instead of just doing some magic. No tool can decide how much security you need. You are the smart person to make that decision for your system or organization.\nBaselining and priorities Setting priorities for system hardening is better than just setting fixed thresholds to your security posture. And yes, it helps when you have at least some baseline. This way you get the feeling of getting closer to the end goal. Priority setting in Lynis Enterprise we do by calculating system risks and ordering (from bad to good). While the number alone doesn\u0026rsquo;t say much, you can compare it with other systems. Users of the community version of Lynis know this as the Hardening Index. And it works! I\u0026rsquo;ve seen people battling who could get the highest score during workshops.\nSecurity policy If you truly want to say that you reached the right level of security, then it should be documented. Not in a text file on just a single system. It should be part of your security policy. You can even create a policy per operating system, stating the minimum required steps to be taken. This can be extended by requirements for each role a system has. You could use instructions like \u0026ldquo;All production systems should have no installed compiler installed unless required for business purposes. In such case, access to the compiler is only allowed to a particular process (or user) and documented.\u0026rdquo;\nConclusion The answer to the simple question we started with really depends on your environment or organization. Still there are steps that every organization should take, like having a clear inventory of systems used and stored in a central database. The next level of knowledge to gain is the risks, threats, and vulnerabilities to those systems and the business.\nThree last tips to get started (Just Do It!):\nPrioritize systems based on known risks and threats first Start implementing with the quick wins Measure defects from your security policies Thoughts regarding this article? Let it know!\n","permalink":"https://linux-audit.com/how-much-system-hardening-should-you-do/","tags":["faq","linux","system hardening"],"title":"How much system hardening should you do?"},{"categories":["Linux"],"contents":"The biggest open source company is nowadays Red Hat. It is known for its contributions to many open source projects, including the Linux kernel itself. Less known is that Red Hat is involved in different Linux distributions, directly or indirectly.\nFedora Fedora has received many updates and individual releases over the years. It is a playground for new functionality. Often new technology is found here. It can be compared with other distributions like Arch Linux, except that it is slightly less aggressive in deploying the latest software components for everything.\nCommunity driven Short release cycles (6 months) Focus on features and new technology Common on desktop The difference between Fedora and other distributions is the corporate support by Red Hat. That means that professional developers can work on projects that are first tested in Fedora. A lot of these components may then also be picked up by other distributions. It also feeds the RHEL product. Everything that is considered to be stable and useful for demanding enterprises, might be moved in phases towards the RHEL distribution.\nRed Hat Enterprise Linux (RHEL) The Enterprise product of Red Hat is named RHEL for short. The main difference with Fedora is that is focused on companies which prefer stability. The most business-critical services are deployed on this platform. Battle-tested components might finally end up in this distribution.\nFocus on stability Supported by Red Hat Paid license Common on server For companies relying on the stability of a Linux distribution, might want to go for this distribution. It is also common to see a split: some systems runs RHEL, less business-critical systems run CentOS.\nCentOS CentOS is a spin-off of RHEL. It is based on the same code base. It has recompiled all the source packages in it, making it effectively a very similar system. CentOS is great if you like the stability of RHEL and want to reduce your costs.\nBased on RHEL Community driven Focus on stability Free This option might be less suitable for business-critical services as it isn\u0026rsquo;t officially supported by Red Hat. Also if you prefer recent software packages, CentOS (like RHEL) might not always be the best option.\nGot some other clear differences? Let it know it in the comments.\n","permalink":"https://linux-audit.com/difference-between-centos-fedora-rhel/","tags":["comparison","linux"],"title":"Difference between CentOS, Fedora, and RHEL"},{"categories":["Malware","Threats"],"contents":"Times are changing when it comes to Linux malware. Since a long time we had backdoors, PHP shells, and even rootkits. But it won\u0026rsquo;t take long that ransomware will catch up on the Linux platform. We hope you are reading this to counter the threat, not because it is already too late.\nRansomware invasion Ransomware is a little devil. It encrypts your valuable data and protects it with a generated key. This key is then forwarded to the maker of the ransomware, and then it is safeguarded. The key is released upon payment, together with a decryption utility. And surprisingly, the bad guys will deliver each time. This way they know people will keep paying for ransomware intrusions.\nThe sudden spike in ransomware is most likely caused by different factors. In other words, each individual factor was an existing technology. Combined they make it a good recipe for evildoers. So is there the increase of data and companies consider that one of their biggest assets now. The spread of internet technology and lowering prices helped. And if you add Bitcoin into the mix, you have anonymous payments. This combination makes it ideal to infect people, encrypt their precious data, and finally ask for them to pay in Bitcoins.\nWhy Linux? In every market where there is money to make, there will be more competition over time. Until there is a point that everyone has to drop prices, or go out of business (or both). The Microsoft Windows platform already had its fierce competition. Now macOS and Linux are next.\nA proof of concept (PoC) is already available for Linux. It is called BashCrypt and comes with everything you need to set up a ransomware infrastructure. It includes the code you have to run on the intruded system and also the code for the server side, to receive status updates and payments.\nDefending against ransomware Staying clean of ransomware is hard, especially if there are many people working in your company. We all (should) know by know that you don\u0026rsquo;t open up strange attachments. But it still happens. User awareness is key and it is something we will have to keep doing.\nIf you have a Linux server which acts as a mail server for your environment, then it makes sense to test some ransomware samples and see if they are detected by the existing anti-virus solution. If not, that is a first place to improve. You might want to make the jump from free open source anti-virus like ClamAV, and add a second scanner on top of it.\nIn the event you became a victim of ransomware, you have two options: pay, or restore. Giving money to bad guys is actually a bad thing to do. It keeps financing them, resulting in an increase of ransomware. Better is to restore your data. So make sure you have good backups, and check them regularly. Why wait? Do check it now and see if you can restore some of your most important data.\nStay safe and till the next post.\nGot any experience with ransomware on Linux? Let it know!\n","permalink":"https://linux-audit.com/malware/linux-and-the-rise-of-ransomware/","tags":["linux","malware","ransomware"],"title":"Linux and rise of Ransomware"},{"categories":["Identity and Access Management"],"contents":"We get often the question what one should do with unused users on Linux. Everyone who looked in the /etc/passwd file will recognize them, strange usernames. A great example is UUCP, or Unix-to-Unix Copy. Once used for communication on direct lines, now another piece of history in our password files.\nThe Options Before we make any decision on dealing with unused Linux accounts, we should look at the most obvious choices we have. The options include:\nKeep them Disable Delete Keep the account The first option is the easiest. Simply take no action and keep the users in the file. While this is a totally valid strategy, it might not be the best option. One reason for this is pollution in the passwd file. It might give room for \u0026ldquo;hidden\u0026rdquo; accounts, especially between legitimate non-personal service accounts (like www-data).\nDisable the account Next option is to disable each account that is not used. Disabling can be done in different ways.\nOption 1: Change the password Changing a password is done with the passwd utility. To change the password of another user, you will need root permissions.\npasswd username\nThis option is very basic. Although it does make sure a user can not log in without knowing the new password, public key authentication via SSH is still possible.\nOption 2: Set an expiry date on the account A slightly more advanced option is to mark the account as expired. This way the user can\u0026rsquo;t log in with a known password, nor with public key authentication.\nchage --expiredate 0 username\nThe beauty of this is that accounts can be disabled, without their password being touched. The second positive thing to remark is the clear message an user gets when trying to log in.\nYour account has expired; please contact your system administrator\nIf we would like to activate the account again, we give it an expire date of -1, which means never.\nchage --expiredate -1 username\nA few tips\nUse -E as short version of -expiredate Use chage -l (or -list) to see information about an account Using chage to expire and unlock accounts\nOption 3: Lock the account You can also lock an account with the usermod utility. It is also advised to change the shell to /bin/nologin.\nusermod -lock -expiredate 1 -shell /bin/nologin username\nThis will make impossible to log in, similar to using chage. Unlocking with usermod is also still possible.\nusermod -unlock -expiredate 99999 -shell /bin/bash username\nDelete Account The last option is making serious changes to the system and delete unused accounts. The LSB specification states that some accounts are optional. Distributions usually simply add the accounts, to account for all users. That doesn\u0026rsquo;t mean you should keep them on your systems. For example Arch Linux took serious measures and removed most of the unused accounts by default.\nCheck password files If you feel you are ready for the task, then the first step is check if an account is actually available on the system using getent.\ngetent passwd uucp\nThe getent utility requests entries from the Name Service Switch (NSS) libraries. This includes password files, group files, DNS resolving (including /etc/hosts ), etc.\nCheck processes running as a user If the account exists, we should also check if any process is running under the ID.\nps aux | grep \u0026quot;^uucp\u0026quot;\nIf this reveals any process, the account appears to be still active. If this is a service account, it might be of legitimate use. If it is an old colleague, then it is time to determine what is running and see if it can safely be stopped.\nCheck files owned by a user Next step is making sure that no files are owned by the user account. The easiest way is to use the find command and start searching on the root file system.\nfind / -xdev -user uucp\nBy using -xdev we tell find not to switch to other file systems. It is better to check them manually, before it starts searching all mount points (like your NFS shares!).\nDelete user account When no more files or processes are in use, you can delete the account with the deluser command.\nConclusion Unused Linux accounts are common. This is especially true with most Linux distributions, which deliver service accounts as part of a default installation. With a good approach it is easy to determine which users are still needed. By using password expiration for normal users, we can more easily detect which accounts are no longer needed and remove them.\nArch Linux has a pristine list of service accounts, not much to clean\nHappy hardening!\n","permalink":"https://linux-audit.com/authentication/unused-linux-users-delete-or-keep/","tags":["authentication","etc","passwd","shadow"],"title":"Unused Linux Users: Delete or Keep Them?"},{"categories":["SSH"],"contents":"Introduction into Ed25519 OpenSSH 6.5 added support for Ed25519 as a public key type. It is using an elliptic curve signature scheme, which offers better security than ECDSA and DSA. At the same time, it also has good performance. This type of keys may be used for user and host keys. With this in mind, it is great to be used together with OpenSSH. In this article, we have a look at this new key type.\nDSA or RSA Many forum threads have been created regarding the choice between DSA or RSA. DSA is being limited to 1024 bits, as specified by FIPS 186-2. This is also the default length of ssh-keygen. While the length can be increased, it may not be compatible with all clients. So it is common to see RSA keys, which are often also used for signing. With Ed25519 now available, the usage of both will slowly decrease.\nConfiguring the server The first thing to check is if your current OpenSSH package is up-to-date. You will need at least version 6.5 of OpenSSH.\nssh -V\nRecreate the SSH host keys Next step is creating the keys for the SSH daemon.\ncd /etc/ssh\nOptionally make a copy of the existing host keys\nmkdir backup \u0026amp;\u0026amp; mv ssh_host_* ./backup/\nThen create the key pair using Ed25519.\n# ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N \u0026#39;\u0026#39; -t ed25519 Generating public/private ed25519 key pair. Your identification has been saved in etc/ssh/ssh_host_ed25519_key. Your public key has been saved in etc/ssh/ssh_host_ed25519_key.pub. The key fingerprint is: 96:67:0f:50:8d:16:51:c2:47:9c:4e:85:b4:79:bd:6b root@arch The key\u0026#39;s randomart image is: +--[ED25519 256]--+ | .=X++. | | .+.Bo . | | .. +o . .| | o .. .| | S + . | | . o o .| | . E | | . | | | +-----------------+ Change SSH configuration (server) Next step is changing the /etc/ssh/sshd_config file. Add the new host key type:\nHostKey /etc/ssh/ssh_host_ed25519_key\nRemove any of the other HostKey settings that are defined.\nClient Configuration After configuring the server, it is time to do the client. We have to create a new key first. Make sure that your ssh-keygen is also up-to-date, to support the new key type. Note: the tilde (~) is an alias for your home directory and expanded by your shell.\n$ ssh-keygen -t ed25519 -C \u0026#34;michael@linux-audit.com\u0026#34; Generating public/private ed25519 key pair. Enter file in which to save the key (/home/michael/.ssh/id_ed25519): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/michael/.ssh/id_ed25519. Your public key has been saved in /home/michael/.ssh/id_ed25519.pub. The key fingerprint is: a0:b4:7a:e5:7e:85:45:ff:12:df:ef:aa:12:e4:ad:e0 michael@linux-audit.com The key\u0026#39;s randomart image is: +--[ED25519 256]--+ | | | . | | . . . . | | . o . o o | | o . S= . + . | | . o o + o o .| | . . .. o o . .| | . . E o .| | .. ....o.| +-----------------+ Optional step: Check the key before copying it.\nssh-keygen -l -f ~/.ssh/id_ed25519\nIf that looks good, copy it to the destination host using the ssh-copy-id command.\nssh-copy-id -i ~/.ssh/id_ed25519.pub michael@192.168.1.251\nThen determine if we can log in with it.\n$ ssh -i ~/.ssh/id\\_ed25519 michael@192.168.1.251 Enter passphrase for key \u0026#39;~/.ssh/id\\_ed25519\u0026#39;:` When using this newer type of key, you can configure to use it in your local SSH configuration file (~/.ssh/config). Defining the key file is done with the IdentityFile option.\nHost [name]\nHostName [hostname]\nUser [your-username]\nIdentityFile ~/.ssh/id_ed25519\nIdentitiesOnly yes\n","permalink":"https://linux-audit.com/ssh/using-ed25519-openssh-keys-instead-of-dsa-rsa-ecdsa/","tags":["ed25519","ssh","ssh-copy-id","ssh-keygen","ssh daemon","sshd_config"],"title":"Using Ed25519 for OpenSSH keys (instead of DSA/RSA/ECDSA)"},{"categories":["Kernel"],"contents":"The GNU/Linux kernel powers a lot of systems, from big mainframes to the Android device in your pocket. If you want to achieve more security on your Linux systems, it would make sense to start hardening there, right? While securing the kernel looks easy at first sight, there is more to it than initially meets the eye. We will have a look at some kernel options and how to select the best sysctl values for Linux systems.\nAfter reading this article you know\u0026hellip;\nWhy you should take care of tuning the Linux kernel and apply hardening measures What some of these hardening measures are useful How to use sysctl and make changes to the system How to ensure that sysctl settings are applied after reboots Why invest time in Linux kernel security? There is a lot going on within the Linux kernel. It is like a complicated machine with many small tasks to perform. Most of these tasks are low-level interactions with the hardware, like writing bits to disk or sending data to the video buffer. When we look at it from a security point of view, there are a few areas which can use our attention:\nFile systems Kernel modules Networking Processes Debugging The options to harden the system in these areas depend on the type of kernel you use (monolithic or dynamic). When using a big monolithic kernel, every required module is added during compilation time. For security a good thing. At the same time, it also means less flexibility. For that reason, it is common to see the \u0026ldquo;dynamic\u0026rdquo; kernel being used. This means we can alter the behavior of the kernel during runtime. The most obvious example is loading new kernel modules, to provide more information. Something that is not possible with a monolithic kernel.\nKernel tuning Kernel parameters are one of the things we can change independently of how the kernel was built. By using a system control interface, we can talk to the kernel and read and change some settings. Good to know is that this is a powerful way to influence the behavior of the kernel. So it also comes with some risk. Still, it is definitely worth better understanding several of these areas, to optimize your kernel and have an increased level of performance and security. By knowing the details of these so-called tunables, you can find the best possible sysctl values for your environment.\nThe interface to the kernel parameters is a commonly available utility named sysctl, short for system control. It talks with the Linux system control interface, which allows reading and writing to the available settings. These settings we call key-value pairs. So when referring to a sysctl key, we mean the setting by its name (e.g. vm.panic_on_oom). We can query the values with the sysctl command.\nsysctl -a\nThe output will look something like this:\nOverview of kernel settings\nThe procfs file system Before we dive into some of these kernel settings, it is good to know that on Linux you will need the procfs file system. You will not have to configure this usually, as it will be deployed automatically by your Linux distribution. To know if you have it, simply use mount and search for proc. And most likely it is mounted on /proc.\nproc on /proc type proc (rw,noexec,nosuid,nodev)\nThis /proc mount point is based on a pseudo file system. That means the files in there are not normal files. Instead, they are special file handles for usage by the kernel and the system administrator.\nBy using the cat command, we can see the value of a particular entry in /proc. Some of the available allow also writing, which equals setting a setting. The /proc mount point contains a lot of files. For this article, we have a special interest in the files in /proc/sys, as they reflect the keys used by sysctl.\nMany common system administration utilities are using the /proc file system. Examples are tools like ps, free, top, and watch.\nUnderstanding the structure in /proc/sys When we have another look at the image above, we see several keys starting with \u0026ldquo;kernel.\u0026rdquo;. If we map these against the /proc file system, we will see they are in /proc/sys/kernel.\nSo when looking in /proc/sys, we can see that the main categories are:\nabi (application binary interface) debug (debugging) dev (devices) fs (file systems) kernel (kernel) net (network) vm (virtual memory) Using sysctl: keys and values Now comes the tricky part: understanding all the keys as part of these categories. Some of them will give a good idea, like /proc/sys/kernel/ctrl-alt-del, which determines how to act on this particular key combination. Most other keys remain cryptic for most of us. For that reason, there is a sysctl overview.\nAnother source is to look at the kernel documentation project . If we use the /proc/sys/kernel/ctrl-alt-del example, that means we can see its current value by catting the file.\ncat /proc/sys/kernel/ctrl-alt-del\nOn our test system this returns a value of zero. A first guess would make us think that it is disabled. To be sure, we look in the kernel documentation and see:\nctrl-alt-del: When the value in this file is 0, ctrl-alt-del is trapped and sent to the init(1) program to handle a graceful restart. When, however, the value is \u0026amp;gt; 0, Linux\u0026#39;s reaction to a Vulcan Nerve Pinch (tm) will be an immediate reboot, without even syncing its dirty buffers. Note: when a program (like dosemu) has the keyboard in \u0026#39;raw\u0026#39; mode, the ctrl-alt-del is intercepted by the program before it ever reaches the kernel tty layer, and it\u0026#39;s up to the program to decide what to do with it. So in other words, the key combination is not disabled. It is actually intercepted and the init process can make a decision on how to handle it. Now the interesting part is that Linux distributions will handle this differently, especially now with systemd being common. Before the action to take by the init process would be in /etc/inittab .\nca::ctrlaltdel:/sbin/shutdown -t3 -r now\nWith systems using systemd, it will be depending on a target file (/usr/lib/systemd/system/ctrl-alt-del.target) and might look like this:\n# cat /usr/lib/systemd/system/ctrl-alt-del.target # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Reboot Documentation=man:systemd.special(7) DefaultDependencies=no Requires=systemd-reboot.service After=systemd-reboot.service AllowIsolate=yes JobTimeoutSec=30min JobTimeoutAction=reboot-force [Install] Alias=ctrl-alt-del.target As you can see, a fairly simple kernel sysctl key, might require some more reading, before making a decision on the best sysctl value. So if you really want to disable the Ctrl+Alt+Del combination, you might want to change this systemd target file. Instead of calling the systemd-reboot.service, it should do nothing, so it truly ignores it.\nChanging kernel settings If you want to change a specific kernel setting, then \u0026ldquo;write\u0026rdquo; it to the file. The simple way of doing this is by using the echo command. So to set the ctrl-alt-del key from our previous example to 0 (zero) as well:\necho 1 \u0026gt; /proc/sys/kernel/ctrl-alt-del\nSimple as that, and easy to script. Even deployment with a configuration management tool like Ansible or Puppet can help in configuring your systems properly. That is, if you fully understood what to tune and why.\nWhy use sysctl instead of /proc? So although it is possible to change runtime behavior of the kernel via the pseudo file system /proc, the sysctl utility has some advantages. The first one is that it can save settings in a file, which gets used upon a reboot. The second reason is that the tool can quickly gather all settings and display it (even with regular expressions!).\nMaking temporary changes It is good to know that changing a kernel setting will not change it permanently. If you use sysctl to set a value, that is similar to making a change to a file in the /proc directory. In our earlier example, we changed the ctrl-alt-del setting and gave it a value of zero (0). Do to this with sysctl, we add the category followed by a dot and the key. As this setting is part of the category kernel, the related command would be:\nsysctl -w kernel.ctrl-alt-del=0\nTo read the value, we use sysctl again.\nsysctl kernel.ctrl-alt-del\nIt should now display the previously set value. Notice that some values can\u0026rsquo;t be set, as they are special counters or special purpose keys. So know this rule does not apply to all settings.\nMaking permanent changes Configuration file: /etc/sysctl.conf Sysctl can store configured settings in /etc/sysctl.conf . These settings are applied during boot time.\nConfiguration directory: /etc/sysctl.d Modern Linux distributions don\u0026rsquo;t store all settings into one big file, but split into smaller sections. These individual files are stored in /etc/sysctl.d. Together with sysctl.conf they form the sysctl configuration.\nReloading the sysctl configuration Another way to set a variable is changing the /etc/sysctl.conf and reload it. This way we can check if the setting is changed, and will be properly applied next boot as well.\nsysctl -p /etc/sysctl.conf\nBy default /etc/sysctl.conf is used. In this case, the file name was not needed. For testing purposes, it could make sense to use a different file. Then when everything works as expected, make the changes to /etc/sysctl.conf or better, a separate file in /etc/sysctl.d.\nShowing keys with regular expressions To see all keys, we use the -a or -all parameter. If we combine this with a pattern, we can quickly display all items containing a particular word.\nsysctl -a --pattern \u0026quot;ipv6\u0026quot;\nIf you are into regular expressions, you can go wild ;-)\nDid you learn something from this article? Great! Got any questions or suggestions? Let it know!\n","permalink":"https://linux-audit.com/system-hardening/linux-hardening-with-sysctl/","tags":["hardening","kernel","kernel hardening","kernel modules","linux","server hardening","sysctl","system hardening"],"title":"Linux hardening with sysctl settings"},{"categories":["Vulnerabilities"],"contents":"As the author of Lynis, we hear often the question: It is like Nessus, right? It seems that everything is compared with Nessus, especially when it comes to Linux security. Surprise, it is not. Let\u0026rsquo;s get things straight, and talk about the benefits of both.\nVulnerability Scanning Scanners like Nessus and OpenVAS are great tools. You drop a system in the network and start scanning. The scanner then usually starts with a ping sweep to detect which systems are alive and providing services. Next step is determining these services, so they can be followed up with more in-depth tests.\nVulnerability scanners are comprehensive and at the same time \u0026ldquo;stupid\u0026rdquo;. They don\u0026rsquo;t exactly know what is on a system, so they have to try. No surprise that your log files are filled with attempts for non-existing files, or ports that are closed.\nVulnerability Classification If vulnerability scanning is level 1 of the game, then vulnerability classification is the next one. After the scanner is done with scanning, you get a big list of (possible) findings.\nLet\u0026rsquo;s say you are running Apache 2.4.6. So the vulnerability scanner tells you about this vulnerability , with the advice to start patching. You might be surprised and thinking that your software patching process should have covered this. Not much later you discover it was actually patched, with a security update from your Linux distribution. The scanner actually provided you with a \u0026ldquo;false positive\u0026rdquo;. It thought you were vulnerable because it didn\u0026rsquo;t know better. The reason for this to happen is simple, it used the version from a banner displayed by the Apache web server.\nBenefits of vulnerability scanners Besides a few downsides of vulnerability scanning, the simplicity of deployment is a benefit. No installation is needed on the systems, and it can do a lot of tests on all kind of devices.\nLinux Auditing So back to Lynis and the comparison with Nessus. Lynis is a security auditing tool for systems running Linux or a UNIX derivative like *BSD and Mac OS X. It is host-based, meaning you have to run it on the system itself. Because you are doing so, it knows almost everything happening on the system. From your NTP servers used for time synchronization, up to running processes, and what packages you have installed.\nThe goal of an auditing tool is completely different to vulnerabilities: it does a health check of the system. If we compare it with an apple: a vulnerability scanner looks for bad spots on the outside, the auditing tool looks from the core of the apple towards the outside. So auditing tools like Lynis go much more in-depth. The cost is that it has to run on the system and that it has to be tailored to the platform being scanned. No surprise that we didn\u0026rsquo;t work on a Windows version yet, as that is a completely different league.\nThat the main goal of an auditing tool is to perform a health check, doesn\u0026rsquo;t mean it is limited to that. It can actually also find vulnerabilities. Lynis can detect outdated packages, without having to maintain a database of \u0026ldquo;bad\u0026rdquo; versions. Instead, it uses the package manager to get these details. It is more accurate and usually up-to-date than someone having to update a list manually.\nWhat if we combine things? If you want a true checkup of your network, you want to combine generic vulnerability scanning, with an in-depth system audit. This way you can get the best out of both solutions. The vulnerability scanner searches continuously for bad spots on the outside. The auditing tool helps you with system hardening from the inside. Do this on an ongoing basis, and you have already some corner pieces of the security puzzle in place.\nFor now, keep on scanning with both types of scanners. But remember, Nessus and OpenVAS are great tools, but you want to extend it with in-depth scanner Lynis.\nHappy hardening!\n","permalink":"https://linux-audit.com/vulnerabilities/why-auditing-and-vulnerability-scanning-are-different-things/","tags":["lynis","openvas","vulnerabilities","vulnerability management"],"title":"Why Auditing and Vulnerability Scanning are Different Things"},{"categories":["Linux"],"contents":"Linux security blogs Finding blogs dedicated to Linux security can be challenging. We consider Linux Audit to be the best blog about Linux security and securing your system. Not very humble, but actually there aren\u0026rsquo;t that many blogs about the topic. That is why we made an effort to seek the best and most influential blogs that at least cover some Linux security.\nWhat makes it influential? It should have quality articles, updated regularly, and tailored to Linux or UNIX security. The countless \u0026ldquo;How to\u0026rdquo; websites are skipped.\nMonths of searching and reading resulted in a list of blogs, sorted by category. If you are interested in the developments on Linux security, add them to your RSS feed reader. For some subjects we couldn\u0026rsquo;t find a specialized blog yet. In that case, we added some filler articles from our own Linux Audit blog. If you have a better replacement, we love to hear.\nLinux developers Some of the most influential people to Linux security are those who do work on it. While they don\u0026rsquo;t specifically specialize in one subject, it wouldn\u0026rsquo;t be fair to leave them out.\nKees Cook Kees currently works for Ubuntu. He does kernel development and focuses on several areas related to security. One of these areas is seccomp, a framework to restrict the available system calls to processes. Very useful for sandbox implementations, or restricting your web browser. The Chrome browser is known to have implemented it.\nKees has its own blog and can be found on X .\nMalware Research One of the names to come in mind for malware research is Lenny Zeltser . He is the creator of the REMnux Linux distribution, which helps you performing malware analysis. He can be found on X well.\nIn the same field, you can\u0026rsquo;t ignore the people behind research group Malware Must Die! , or MMD. They cover a lot Linux related malware and explain on their blog how it works.\nCompliance PCI DSS compliance Many companies have articles about PCI DSS on their website. Unfortunately, I couldn\u0026rsquo;t find a quality blog which covers PCI and Linux in particular.\nSuggested article:\nIn-depth Linux Guide to Achieve PCI DSS Compliance and Certification System Hardening No particular blog is known to specifically talk a lot about this subject. Here is an alternative post we created:\nLinux server hardening: most important steps to secure systems SELinux When thinking about SELinux, two names come directly to mind: Paul Moore and Dan Walsh. Both have their own blog, usually talking about the developments on SELinux. Paul includes the yearly \u0026ldquo;State of SELinux\u0026rdquo; presentations on his personal site, which is giving on the Linux security summit.\nBlog by Dan Walsh Blog by Paul Moore Others Some blogs cover more generic subjects. We have found at least the blog of Robert Penz , who writes on a regular basis about Linux, or IT security. Enough to consider them also specialized enough to make it to our reader list.\nThen there is Major Hayden , a system engineer focused on automation and security. He writes about a wide range of topics, usually sharing problems he encounters while doing his work. Great for those encountering the same issues, and don\u0026rsquo;t want to spend the same amount of time he had to.\nGot another blog about Linux security that was not mentioned yet? Let it know!\n","permalink":"https://linux-audit.com/the-most-influential-linux-security-blogs/","tags":["blog","linux security","selinux"],"title":"The Most Influential Linux Security Blogs"},{"categories":["File Systems"],"contents":"The history of hidden files Ever wondered why there are files on your Linux system, starting with a dot? The short answer: they are shortcuts. The story begins many years ago when the first file systems were created on UNIX. To allow easy navigation, a single file with a dot (.) was added to each directory. Secondly, a double dot file (..) was added to easily move up in the directory structure. As these files had no real data in them, a quick hack was added to the ls binary.\nThe \u0026ldquo;hack\u0026rdquo; The change made to the ls binary involved checking for the first character. If that was a dot, it should be ignored. And it worked great!\nSomething which was not anticipated is what we now know as a hidden file. If you create a file starting with a dot, it will become a file on disk, but which is not displayed by default. This behavior is the result of the earlier hack applied to the ls binary.\nBig difference when seeing all files\nAre Hidden Files Bad? So with all these hidden files, we might wonder if their purpose is good or bad. After all, it can declutter your home directory, by showing less files. While this is true, the opposite is true as well. Most utilities create hidden files, and might not really clean things up when unneeded. With utilities scanning your home directory, things slowly will cost more and more time.\nMore Details In a Google+ post by Rob Pike, A lesson in shortcuts , the more detailed rationale behind the dot files.\nLong ago, as the design of the Unix file system was being worked out, the entries . and .. appeared, to make navigation easier. I\u0026rsquo;m not sure but I believe .. went in during the Version 2 rewrite, when the file system became hierarchical (it had a very different structure early on). When one typed ls, however, these files appeared, so either Ken or Dennis added a simple test to the program. It was in assembler then, but the code in question was equivalent to something like this:\nif (name[0] == \u0026lsquo;.\u0026rsquo;) continue;\nThis statement was a little shorter than what it should have been, which is if (strcmp(name, \u0026ldquo;.\u0026rdquo;) == 0 || strcmp(name, \u0026ldquo;..\u0026rdquo;) == 0) continue;\nBut hey, it was easy. Two things resulted. First, a bad precedent was set. A lot of other lazy programmers introduced bugs by making the same simplification. Actual files beginning with periods are often skipped when they should be counted. Second, and much worse, the idea of a \u0026ldquo;hidden\u0026rdquo; or \u0026ldquo;dot\u0026rdquo; file was created. As a consequence, more lazy programmers started dropping files into everyone\u0026rsquo;s home directory. I don\u0026rsquo;t have all that much stuff installed on the machine I\u0026rsquo;m using to type this, but my home directory has about a hundred dot files and I don\u0026rsquo;t even know what most of them are or whether they\u0026rsquo;re still needed. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.\nI\u0026rsquo;m pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.\nSo what do you think? Are dot files evil or good?\n","permalink":"https://linux-audit.com/linux-history-how-dot-files-became-hidden-files/","tags":["file system","ls"],"title":"Linux History: How Dot Files Became Hidden Files"},{"categories":["Auditing"],"contents":"Why both look the same, yet have subtle differences\nWhen talking about auditing, I see that most technical people immediately think about vulnerability scanning. While they definitely have things in common, there are also a lot of minor differences. In this blog post I will show them, and also share how technical auditing and vulnerability scanning can work together.\nSimilarities and Differences Let\u0026rsquo;s first determine what makes technical auditing and vulnerability scanning look similar. First of all, both processes have a technical focus with the goal to discover. The output of both is usually a list of issues. The ones performing the tests have both a technical background. But then things get different.\nWhen we talk about technical auditing, we mean performing an in-depth health check of a system. A technical audit looks at different areas of the system, to determine how well it is configured. Vulnerability scanning on the other has the main purpose to detect software flaws. It is often used by penetration testers and other security professionals, to determine how well a system is patched.\nOpenVAS and Lynis In the field of Linux systems, let\u0026rsquo;s compare OpenVAS and Lynis. The first is an open source vulnerability scanner, the latter an open source auditing scanner. Both tools have the purpose to find weaknesses on the system. Where OpenVAS does a wide range of tests from the network, Lynis runs on the host itself. Both tools will find different findings, depending on the detected services.\nIf you would only run OpenVAS, you might it detected some services running, like a web server. It will then perform a set of tests against the HTTP or HTTPS port and reveal its findings. Such findings could be weak ciphers used in the SSL/TLS configuration. While that is a good thing, it might be totally missing that your system time is not properly synced. These kind of things can be detected by Lynis. So in the end it does not make sense to compare vulnerability scanners and auditing tools, as their focus is different. If you would compare something, then take Nessus and OpenVAS, and compare those. For Lynis you could compare it with OpenSCAP.\nOverview Similar Technical focus Find weaknesses Different Audit performs health check, vulnerability scan checks for software weaknesses Audit can be more generic, vulnerability scan focuses on software ","permalink":"https://linux-audit.com/vulnerabilities/difference-between-auditing-and-vulnerability-scanning/","tags":["audit","vulnerability scan"],"title":"The Difference Between Auditing and Vulnerability Scanning"},{"categories":["Software Development"],"contents":"Last month the Core Infrastructure Initiative, or CII, launched their CII best practices project (now OpenSSF Best Practices Badge Program ). Its primary goal is to gamify the process of building more secure software. Let\u0026rsquo;s have a look at the project, and how it can help.\nOpen Source and Security If we look in the open source world of software, we see that many projects were created by volunteers. While doing this voluntary, this doesn\u0026rsquo;t say anything about the quality of the project. After all, half of the internet exists because of these small, yet powerful utilities. I personally created two projects myself: Rootkit Hunter (rkhunter) to detect malware, and Lynis to perform a security audit on Linux and UNIX systems. While these tools are focused on security, it is definitely not simple to make software itself secure. This is where a project of CII comes in, to provide a checklist of items to enhance the project and its quality.\nWhy Best Practices? The last few years a lot of serious vulnerabilities have been discovered, like Hearthbleed, POODLE, and GHOST. The supporting organization behind CII is the Linux Foundation. They support projects which are considered to be critical for the open source community, and our society in general. With the core infrastructure initiative, they go a step further by allowing all software projects to perform a self-audit.\nBest practices are considered to be steps and actions that are commonly accepted as good things to do. For example, when dealing with sensitive data on your website, that you make your website available via HTTPS. This ensures that the transfer of data is encrypted, increasing security for your visitors. As you can imagine, there is usually a huge list of best practices available for most activities, including software development.\nExample: Lynis For our auditing tool Lynis we applied to get a badge as well, to show that we are committed to do the right things. We are a security tool after all, right? So a few weeks ago we started the process. The first step is creating an account and providing some basic details about the project, like the name and website. From there on you directly move on to the specific best practices.\nThe CII project performs some testing, but mostly relies on you answering the questions honestly. Most questions might sound very obvious, like if there is a page which describes what the tool does. Still, as part of daily research I come across projects which don\u0026rsquo;t clearly describe that. So it is a good thing that these basic questions are asked to trigger the developer to think outside the technical boundaries of the project. After all, software is created for the users.\nDoing secure software development means more than checking for buffer overflows. So CII asks questions about:\nDocumentation (in English) Usage of HTTPS Selected license Version numbering Changelog As you can see, from these first five items, only one is related to security. The others are more generic ways of testing the quality of a project. Not having a changelog file to describe the changes, is something to be considered bad by users. After all, you want to see what is changed and in what particular version. We went actually a step further and renamed our \u0026ldquo;CHANGELOG\u0026rdquo; to \u0026ldquo;CHANGELOG.md\u0026rdquo;, and added markdown to it. This way people can still read the log like before, but it is also rendered by sites like GitHub. A more visual attractive log file, and another improvement for quality of the project.\nThe best practices project has the following categories of tests:\nBasics Change Control Reporting Quality Security Analysis Future By following each category, a list of questions and requirements is displayed. It is up to the \u0026ldquo;auditor\u0026rdquo; to provide an answer, or define if the requirement is met. In some cases you can mark an item as non-applicable and describe why that is. If you satisfied all the requirements, you are rewarded with a badge.\nLessons Learned During the process of obtaining the CII best practices trophy badge, we actually learned more about our own project. For example, we are very strict when it comes to simplicity, as we want to make our software easy to use. On those areas, we had multiple questions which were easy to score. When it comes to code quality, things became a little bit harder to answer. The CII project assumes that most software projects are written with C++, or languages like Python. This is true, except for Lynis. With shell script things are a little bit different, so we had to answer to the best of our knowledge.\nA big improvement, which we already had under discussion, was the usage of unit tests. With our new Lynis software development kit , we introduced quality testing (e.g. unit tests, code linting). Especially regarding unit testing we are now looking to improve in this area, to do more tests. Also we trigger these unit tests now automatically, when a pull request comes in. This is done with Travis CI , and ensures that the quality of the code is as high as possible.\nConclusion What I like the most of the CII best practices project is the same approach it has like Lynis. Both measure quality and the next possible actions to take. The CII best practices project is therefore a great addition to the world of open source software. It helps software projects to question themselves if they are doing the right things. Having good functionality in your software project is a good start. What makes a project truly great, is when users have a great experience along the way. From the first time they visited the project website, to achieving their goals with the software. Getting everything right, including the \u0026ldquo;invisible\u0026rdquo; things, is difficult for most developers. The best practices project helps to make them visible, so the developer can improve the open source project step by step. This results in higher quality software, benefiting all of us.\n","permalink":"https://linux-audit.com/software/secure-software-development-cii-best-practices/","tags":["software","software development","software vulnerabilities"],"title":"Secure Software Development: CII Best Practices"},{"categories":["Authentication"],"contents":"The first thing you might see when connecting to a Linux machine, is a login banner. Some systems use the default, others have put some serious work into it. Think of great forms of ASCII art, or a lot of impressive text. You might be surprised to learn the real reasons for having a banner in the first place. In this article we will discuss the purpose, and determine how we can improve the quality of our login banners.\nReasons for using login banners Most administrators don\u0026rsquo;t put a lot of thought in the banners, like the one used for SSH connections. Some welcome anyone who connects, others filled it with background details of the system. Those with time on their hands, might have added goofy textual graphics. So what is the reason for showing someone a banner in the first place?\nScare hackers? If you ask this question about the purpose, most will say it is to scare away unauthorized visitors. While it might work for some malicious users, most of them use automated scripts. In such case the banner is not even displayed to a human, which defeats its purpose. So if that is not the real reason, there is not much else, right?\nProvide information? Well, if you simply want to be informative about the system, you definitely can do this in a banner. Like sharing what system hostname or IP address you are connecting to. It might be a good confirmation that you connected to the right system. But be careful, you might give away too much information. This is called \u0026ldquo;information disclosure\u0026rdquo;, and makes it easier for attackers to find the information they are looking for. It is similar to provide program names and versions. So being informative can be an option, but it definitely not the main reason.\nLegal? All these mentioned reasons are not the primary goal of a banner. It is about legal and privacy. First, it is to tell upfront that only authorized users are allowed to move forward with the authentication procedure. This way a human connecting manually to a system, is told (friendly) to leave if they shouldn\u0026rsquo;t be there. More interesting is the privacy part. The banner is there to strip away any privacy rights users have on the system. What, removing all privacy for users? Yes, that is correct.\nWith a banner, you warn legitimate users about the possibility of system monitoring and privacy invasion. Of course, it is not like you want to gather their most intimate secrets. But you might have applied automated monitoring and snooping. Like storing every single command they execute, or files they accessed. This already invades privacy, and should be shared with the user.\nThe usage of banners is similar to warning signs, like for video surveillance, or other defensive measures. This way users can\u0026rsquo;t complain they are being watched. With a banner or warning sign you also gain the effect of preventing things from happening. We also call this is deterrent measure, which helps users not crossing the line.\nFor Linux systems, you can define banners on common services like FTP and SSH. You could also add a banner to the login page on a web application, to make clear what happens after logging in.\nGood versus bad banners With these insights in mind, it is a lot easier to come up with a good banner. So here are some tips:\nDon\u0026rsquo;t Welcome the user Provide information only authorized people should know Share system resources or performance Do Share that only authorized people and services are allowed to proceed Explain that monitoring is active on the system Add a line stating that by proceeding, you accept to the terms Now the legal system varies a lot in the world. So for exact wording, it is advised to contact your legal department, if you have any. If you don\u0026rsquo;t have access to a legal person, continue reading.\nExample banner Here is a list of words that you would generally expect to be in the banner:\naccess (by accessing this machine) audit (this system is audited by means of automatic and manual monitoring) accept (by proceeding, you accept the contents of this banner) authorized or unauthorized (this system is only available for authorized users) enforce (policies are enforced to monitor this system) law (unauthorized access will be reported to law agencies) legal (we will take legal measures) monitor (this system is monitored) private / prohibited / restricted privacy (no privacy is guaranteed as this system will be monitored) proceed (by proceeding\u0026hellip;) subject terms Banners of government or other restricted systems will typically have many of these terms in their banners. Use a banner that matches best with the typical audience that will use your service. Verify it meets the do\u0026rsquo;s in this article, and at the same time is not filling up screens or scare away your real users.\nWant to scan your Linux banner? Have a look at Lynis, it will scan files like /etc/issue and /etc/issue.net for common legal words.\n","permalink":"https://linux-audit.com/the-real-purpose-of-login-banners-on-linux/","tags":["authentication","login","privacy","ssh"],"title":"The real purpose of login banners (on Linux)"},{"categories":["Software"],"contents":"Compilers and security Compilers can be the gateway for an attacker. By misusing a possible weakness in your system(s), a compiler is often used to build the related exploit code. One way to prevent this is to determine what compilers are installed and remove (or restrict) them.\nComparing Installed Packages and Compilers One way to audit the system is creating a list of common compilers and packages, then match these with the installed packages.\nCommon compilers Some of the tools found related to compilers are:\ncc gcc go make Programming languages Every interpreter could be abused as well. Especially on systems running network services, like a mail or web server. Making a \u0026ldquo;power tool\u0026rdquo; available to shell users, should be carefully considered. Such user should ideally have no access to Perl, Python, and other languages. If it can be restricted, this will be another piece of system hardening.\nPerl PHP Python Perform an Audit Against Package Database To automate things a little bit further, we can also query the package database. Here are some snippets you can use to determine which compilers are installed, with their respective package.\nNote: package managers have minor differences between each version. Run the commands manually as well to see if things work properly and give the expected output.\nArch Linux #!/bin/sh # Parse pacman output and determine compiler for I in `pacman -Q | awk \u0026#39;{ print $1 }\u0026#39;`; do IS_COMPILER=`pacman -Qi $I | grep -i \u0026#34;compil\u0026#34;` if [ ! \u0026#34;${IS_COMPILER}\u0026#34; = \u0026#34;\u0026#34; ]; then echo $I; fi done CentOS and RHEL With YUM, we can get the information from each installed package. For that we have to loop through all installed packages, query the information, and finally see if it contains something like \u0026ldquo;compiler\u0026rdquo; or \u0026ldquo;compilation\u0026rdquo; in it.\n#!/bin/sh # With the YUM package manager it is harder to retrieve the compilers which are installed. # Got a better suggestion? for I in `yum -q -C list installed | awk \u0026#39;{ print $1 }\u0026#39; | grep -v \u0026#34;Installed\u0026#34;`; do IS_COMPILER=`yum info ${I} | egrep -i \u0026#34;compil\u0026#34;` if [ ! \u0026#34;${IS_COMPILER}\u0026#34; = \u0026#34;\u0026#34; ]; then echo $I; fi done Debian and Ubuntu For Debian and Ubuntu systems no script is needed. Simply use the dpkg command to query the correct information, and grep it.\ndpkg -l | grep -i \u0026quot;compil\u0026quot;\nIf you just want the package names, display the second column.\nGot other tips to find installed compilers on Linux? Let it know!\n","permalink":"https://linux-audit.com/software/audit-installed-compilers-and-their-packages/","tags":["compiler","dpkg","packages","package manager","pacman"],"title":"Audit installed compilers and their packages"},{"categories":["Vulnerabilities"],"contents":"Our digital world is full of hardware and software components. The big difference between the two is the quality. When hardware ships with defects, people will return it and talk badly about it. For software it is fine if things are not perfect from the beginning. It can be improved upon in steps, until most of its users are happy with it. Developers of this software often are some level of pressure. We already know that most of the security vulnerabilities are caused by proper training or lack of quality testing. And even then, it is hard to get everything right.\nWhen we bring a new piece of software into our world, like our notebook or IT environment, we increased our risks. Another target to be exploited. To combat this, we can ensure our software patch management is done properly. The other common option is to actually scan for vulnerabilities and dealing with them, which is vulnerability management.\nWe get a lot of demo requests for our software, and we take the personal approach. Before we share a trial, we want to learn a little bit about our potential customer. Not to nag them, but to better understand if we are both a good match. Very often we tell such requester we are not a good match. In most of these cases, this is because they are looking for purely a vulnerability scanner. If you worked with Lynis or Lynis Enterprise, you know we perform security audits. Deep health checks on your servers, of which vulnerabilities is just a part.\nPain Points With all the emails we received over the years, we found out this is the top 3 of \u0026ldquo;pain points\u0026rdquo;:\nPatching - Customer finds it difficult to get a good software patching process in place, or properly work with it. Next step - Customer does not know how to continue at different stages. Like determining what can be done next. Think if of how to move forward with system hardening for further improvement and reduce risk internally and for their customers. Validation - Be able to validate the effectiveness of patching and hardening efforts, and properly report that. Some people will directly say you will need a vulnerability scanner for that. Your (average) vendor of vulnerability scanner will say that as well. But really, is it a solution? Their focus is broad, to detect many different flaws. At the same time, they leave some stones untouched. Another issue with vulnerability scanners is that they focus on the bad. It is their primary duty to find weaknesses. The more they find, the better the tool is supposed to be. Well, unless you get into false positives.\nNegativity In Security Does Not Work Sometimes we can\u0026rsquo;t avoid expressing a key performance indicator (KPI) as a negative thing, like the number of issues found. But when the number is always bigger than zero, does it really make sense to report on that? This negative expression of security indicators is only depressing people. It is like saying there is no need to put time into it, as it is always in a bad shape. Consider that if you have a few pounds too much and decide to stop exercising at all (because you think it is of no use anyways). As we know, the problem only becomes worse.\nVulnerability scanners have the negative association in them, as their name already focuses on scanning for the \u0026ldquo;weak\u0026rdquo;. It would have been so much better as they were named \u0026ldquo;Security defense checkers\u0026rdquo;. But well, the term is around in the field, and so many people are now programmed to hunt vulnerability. Yet, we consider it the way to depression. That is a shame, as the tools definitely can have a good impact on the general security posture of a company. So time to move away from vulnerability scanners only, and look at the positive side!\nThe Alternative To Vulnerability Management? If you want to leverage vulnerability scanning the best possible extent, you have to combine it. This could be with good reporting, sharing insights, and reward people for doing good things. Don\u0026rsquo;t focus just on the \u0026ldquo;bad\u0026rdquo;, like vulnerability scanners. Instead, set targets and next steps, reward those who comply with them.\nAn interesting alternative to vulnerability management is making it part of a bigger strategy. Combine it into a power tool. What this power tool looks like:\nVulnerability scanner -\u0026gt; Determine any weaknesses, and detect outdated packages Security auditing tool -\u0026gt; Discover health status of environment, show compliance, define next step of improvement Configuration automation tool -\u0026gt; Implement improvement: push configurations and remediations So instead of one single tool, go for a combination of tools, each with their own strengths. While you could still count the number of high-risk vulnerabilities, you have more items to score on. Like the percentage of systems covered by the configuration management system. Or the number of systems scanned in the last week by both the vulnerability scanner and auditing tool.\nLet\u0026rsquo;s start with the last category first.\nConfiguration Management On Linux Systems Regular readers know we focus on Linux and UNIX systems. What makes these platforms great is the flexibility you have to create something that works for your organization. This starts at the stage of doing configuration management. You can do it manually, or create some nifty shell scripts, or decide to go for a configuration management tool. Over the last years, it is Ansible that got a great share in this market, although Puppet still seems to be leading it. But of course don\u0026rsquo;t forget the others, like CFEngine and Chef.\nIn this day and age, automation is key. Where possible you should try to replace yourself with a script. Instead of doing repeating tasks over and over, try to capture them into an easier way of working. Then you can move up to the role at puppeteer, with control over how other pieces in the environment will move and act. Use technologies like PXE boot to deploy systems, Kickstart to install the operating system, and a configuration management tool to maintain the configuration. You should be focusing on the exceptions and the outliers.\nLinux Doctor: Security Auditing Even if you think you have everything under control, things might be wrong. The problem is that we don\u0026rsquo;t know with a proper health check. This is similar to bringing your car for an inspection, or go to the doctor yourself. Linux can also use a good checkup now and then. This helps in detecting the exceptions much quicker. While many things can be automatically checked with your monitoring tooling, some items never might appear on your radar. That is, until you run an in-depth scan. Do you ever check the integrity of your password file, or any users that no longer should be there?\nWant to get convinced there is so much to improve? Give open source tool Lynis a spin.\nAutomatic Software Patching If you have your configuration management, system monitoring, and system auditing in place, it is time to ensure you have a good software patch routine. This means you should also have a policy, to make decisions on how you deal with the ongoing stream of updates. A policy should answer things like how often will you be doing software updating, and how to deal with high-risk security updates. But even if you have a great policy, it is the routine of patching that makes the difference.\nStill too many system administrators are afraid of rolling out patches. For a good reason, as they get blamed when something breaks. For that reason, you should get systems in place to test updates. One thing you could do is set up a small set of virtual machines which gets patched 24/7. When there is a new patch, those systems will receive them first. They mimic production systems, by running a combination of different roles.\nFor example, these patch testing systems could be hosting both the mail server configuration, a web server, together with a database engine. If a patch is released for any of these components (like nginx, Postfix, or MySQL), it will be applied there first. This will also apply for generic updates, like common components as OpenSSL, or glibc. And the system will also need to be rebooted automatically, to ensure kernel updates are tested. Extensive system monitoring will then check to see if the system stays up, even after automatic patch and rebooting.\nThe takeaway is this: automate everything, including those things you rather do manually. Because in the end, you could still do some things manually. And you can, because you saved yourself a lot of time, so you can put in total focus on the things that are rare, or special. This might be your business critical service, or that outdated system which needs to be phased out.\nAutomation is also a great indicator for your security metrics. Instead of focusing on the negative number of vulnerabilities, share what percentage of systems is automatically managed, monitored, or checked.\nVulnerability Scanning This article had the focus on vulnerability scanning. Why the other parts? They can reduce the number of vulnerabilities greatly. Again, automation is key, and the only way to solve newly discovered issues quickly. For that same reason, it is a safe bet to focus on continuous auditing and security monitoring. You can\u0026rsquo;t prevent everything, but you can improve your detection rates. Vulnerability scanning might help here, yet audit tools and event logging will have a greater impact in the end.\nConclusion: stop getting depressed with the negative aspects of security, and vulnerability scanners in particular. Focus on improvement, automation, and reporting. Show the good work we do, leave the bad for the others.\n","permalink":"https://linux-audit.com/vulnerabilities/vulnerability-scanning-the-destiny-to-disappointment/","tags":["auditing","vulnerabilities","vulnerability management","vulnerability scan"],"title":"Vulnerability Scanning: The Destiny to Disappointment?"},{"categories":["Web"],"contents":"On invitation by the Dutch consultancy firm Snow (now SUE), I attended their Snow Unix Event (SUE). It was the third time in a row, with again an impressive lineup of speakers. As I worked previously for the company, I expected no less than that. The theme was about knowledge sharing. That sounds like an invitation to also share some of the biggest insights I learned. Let\u0026rsquo;s start with the HTTP/2 insights by Daniel Stenberg.\nInsights from the cURL developer The first talk of the day was by Daniel Stenberg. If you ever used cURL , you might recognize him name from his personal domain haxx.se. The world is changing, and a tool like cURL must evolve as well. One of those big changes is support for HTTP/2, a fairly new protocol.\nDaniel started off with explaining the importance of HTTP for the internet. We run almost everything on it, and usage grew tremendously. In the last four years, the average website got heavier. For example the number of objects, from 80 growing to 100 objects (HTML, CSS, JavaScript, etc). More impressive was the size of an average web page: four years ago that was about 800 KB, now it is 2300 KB.\nProblem: latency The main issue with \u0026ldquo;current\u0026rdquo; HTTP/1.1 protocol is what Daniel calls the \u0026ldquo;round trip bonanza\u0026rdquo;. A lot of connections are initiated (usually around 40 for the average website), each requesting a single object. The requests itself need to ping-pong between receiver and sender, adding up a lot of delay, as they are queued. The slightest addition of latency makes the current HTTP protocol inefficient. For most users bandwidth is no longer the issue, but this inefficient queuing. Daniel also referred to it as \u0026ldquo;head of line blocking\u0026rdquo;, like choosing in the supermarket the quickest line. The first person in that line will have a huge impact on the rest.\nHTTP/1.1 workarounds Last years people came up with clever ideas, or hacks, to improve the efficiency. Thinks like spriting, where many objects are merged and delivered as one single object. Think of small icons, which are merged into a single PNG. Then there is concatenation (cat *.js \u0026gt; newfile.js), to reduce the number of files. Another example includes embedding those images in your CSS files, with base64 encoding. This is named inlining. Last but not least, Daniel mentioned sharding. As browsers generally allow up to 6 connections per domain, people started to spread their objects on more domains. This effectively allows more connections. With many tabs open in your browser, you can imagine the impact your browser has on the TCP stack of your operating system.\nHTTP History The HTTP protocol itself has a rich history, even though it doesn\u0026rsquo;t have that many versions. It started in 1996 with HTTP/1.0 (RFC 1945), followed by HTTP/1.1 in 1997 (RFC 2068) and an update in 1999 (RFC 2616). Then it was silent for a long time, till 2007 and httpbis refreshed HTTP/1.1.\nThe real big change started in 2009 by the well-known search engine Google. Their protocol SPDY wanted to solve many of the inefficiencies of HTTP/1.1. It was implemented in their data centers and Chrome browser in 2011. In the next year, the work on HTTP/2 started. In 2014 we got a slight revision to HTTP/1.1 (RFC 7230), making room for finally HTTP/2 in 2015 (RFC 7540).\nWhat is HTTP/2? If you create a new protocol, adoption is key. This is especially true for something with the importance of HTTP/2, making a serious change to the web. So Daniel explained the protocol is actually a framing layer, and maintaining a lot. So no changes to the old protocol, nor changing too many things. For example, POST and GET requests will remain. The naming convention (HTTP:// and HTTPS://) will also be unchanged. Another thing pointed out is that HTTP/1.1 will remain for a long time.\nOne of the biggest lessons learned is that protocols should not have too many optional components. HTTP/2 has therefore mostly mandatory items, specifying how it should be implemented. You are compliant with the specification, or not at all. That keeps things simple. For that same reason there is no minor version. The next version will most likely be HTTP/3.\nBinary versus plaintext One of the biggest changes is that the new protocol is no longer plaintext. Telnetting to your web server no longer gives you the typical response. Instead, the protocol is binary, which makes framing much easier. It has also performance benefits, as there are less translations needed. With default support for TLS and compression, things definitely changed. On the question how to do analysis now? Popular tools like Wireshark already have support. More tools for debugging, or security testing, will have to adopt.\nHeaders The new protocol has two frame types, headers and data. Regarding these headers it is interesting to know that they grew over the past years. More data seems to be included, like cookie data. With HTTP/2 these will be compressed as well, especially as there is a lot of repetition involved.\nConnections One of the biggest changes is reusing connections. So data streams are multiplexed now, enhancing data transfers and reducing overhead. Systems together decide on how many parallel streams occur. Another improvement is so-called \u0026ldquo;connection coalescing\u0026rdquo;: if the client detects multiple sites using the same source system, that connections will be \u0026ldquo;unsharded\u0026rdquo; and merged with an existing stream.\nWhere previously HTTP requests were done one by one, in the new protocol they are done at the same time. So no more waiting for the first request to finish, reducing the initial request. To ensure the most optimal data flow occurs, streams can be prioritized by giving them a weight.\nServer push Normally the client does the requests. It requests the first pieces of HTML, then decides what else it will request, like CSS and JavaScript files. With the new protocol that will change. The server will be allowed to make suggestions and push data. That is, if the clients wants to accept that.\nAdoption rates As of April 2016, Daniel shared that Firefox already used 23% over HTTP/2. Regarding HTTPS, that is used for 35% of the traffic. Most browsers already support the new protocol. When it comes to the top 10 million websites, 7% has adopted the new protocol. That increases to 9% for the top 1 million, and even more for the top 500 (19%). Not bad at all, considering that most traffic will go to the last group. One of the big content delivery networks, Akamai, made the switch. The Google bots and Amazon platform are expected to follow soon.\nHow you can adopt Daniel explained that you can easily run HTTP/2 yourself. Make sure to use an up-to-date browser. To see if you are using the new protocol, you can install browser extensions (search for HTTP2 browser indicator).\nIf you are doing things on the server side, it gets a little bit more complicated. First of all you need a web server that supports it. Your version from the repositories might not be up-to-date enough. So have a look at the latest version of litespeed, nginx, or Apache. Newer web server daemons like nghttp2, H2O, Caddy, will definitely have support for it.\nSecond problem is that OpenSSL might be too old, therefore missing APLN support. This stands for application protocol layer negotiation, which defines what protocol to use. Only by running the very latest versions of your Linux distribution, you might be having more luck now. Last but not least, you will have to configure a certificate. Fortunately the Let\u0026rsquo;s Encrypt project makes things easier now, for both configuration and the price tag.\nSo what about HTTP/2 security? As we cover mainly security on this blog, it was interesting to note that a few items around security popped up. One of those was dealing with the some recent attacks like BEAST and CRIME.\nHTTPS A big misconception is that you need HTTPS when running HTTP/2. HTTPS is not mandatory according the specification. However, browsers don\u0026rsquo;t allow it. This means effectively that you will need a SSL/TLS certificate. But if you really want, you can still run the new protocol on port 80. It might be a good thing to have HTTPS available, as it provides privacy and user protection.\nSafe defaults The new protocol does not support things like SSL renegotiation, SSL compression, or any protocol before TLS 1.2 with weak ciphers. So on that level the protocol makes the web a lot safer.\nServer push security The security of server pushing is still vague. In this area more development is needed.\nThe future A few areas are currently still vague. So is the security and implementation of server pushes under development. Also client certificates are not supported yet. Daniel listed this as a possible improvement, together with more tuning of the TCP stack, minor changes to handling of cookies.\nBeyond HTTP/2 it is the goal to slowly drop the legacy of the first HTTP protocol (1.0 and 1.1). HTTP/3 will happen a lot faster, not the 16 years it now took. One of the more interesting developments is QUIC . It is TCP, TLS, and HTTP/2 over UDP! More more head of line blocking, the main issue with using TCP.\nAbout Daniel Stenberg Daniel is from Sweden and works for Mozilla. He still keeps evolving the popular cURL toolkit (curl and libcurl). I had the honor to sit across the table with him and the other speakers. Knowledgeable, friendly and a great sense of humor.\nThanks Daniel for your great talk, and Snow for the invite!\n","permalink":"https://linux-audit.com/web/web-changes-with-http2-performance-and-security/","tags":["apache","certificates","nginx","web"],"title":"How the web changes with HTTP/2: Performance and Security"},{"categories":["Linux"],"contents":"Why that is not a reality, and we might never achieve it.\nLinux gained great popularity over the last 10 years, powering our servers and smartphones. With all the efforts put in creating more secure software, it seems installing security updates will remain a weekly task. Will this ever change?\nSecurity is Hard Properly securing a system means different things for different people. So let\u0026rsquo;s take the assumption that every system has a particular goal, secondly that it should be properly secured. The first one defines what the system should be able to do, like \u0026ldquo;be a web server and provide content\u0026rdquo;. The right amount of security, that is slightly harder to define. We know that at a minimum our security measures should not conflict with the goal, but that\u0026rsquo;s it. How much security is enough?\nBe a Bank? If you are a bank, confidentiality and integrity of data might be more important than availability. While you might argue that the lack availability gives a bad reputation, that is something people will forgive you. Showing incorrect bank statements might be a different story.\nFor most of us, we don\u0026rsquo;t have to harden our servers at the same level of a bank. The question still remains, what security measures are appropriate. Ask that to different security specialists, and they will all give you a different answer.\nRisk Management Some security professionals will directly answer \u0026ldquo;do risk analysis\u0026rdquo;. While this is a good tool to determine the risks involved in running a particular system, it is time-consuming and not an easy task either. If you are a system administrator, do you really know all the risks involved within your business?\nLinux Security Complexity Two things make Linux security complex. The first is what you would count as \u0026ldquo;Linux\u0026rdquo;. You might just count the kernel (GNU/Linux), the kernel plus additional basic system administration commands, or a whole Linux distribution. Depending on that criteria, the last option will give you a lot more to secure.\nThe second part of complexity is the amount of security measures involved. From benchmarks and guides, to tools and utilities. There are so many resources, with even more contradicting advice involved. Sometimes simply no longer correct due to the ongoing changes made to the involved software components.\nA great example of new development is the systemd framework. It is a major change on how Linux systems operate now. At the same time, most of its users won\u0026rsquo;t have any idea what security options are provided, or how to configure them. Sure, there is some security related systemd documentation , but not clear instructions on how to implement it on your systems. Fortunately, there is the systemd section on this website.\nGuides, Guides, and more Guides We often think that choices are a good thing. It has been proven several times that more choice can be harmful, or even result in analysis paralysis . In this last case we might actually end up doing nothing, as we can\u0026rsquo;t decide where to start. For Linux system hardening there is an overwhelming amount of hardening guides available:\nCIS benchmarks NSA hardening guidelines NIST standards Vendor guides GitHub snippets (and gists) Even Linus Himself Dislikes Security The original author of GNU/Linux, Linus Torvalds has a negative opinion about Linux security. No surprise there, as he is known to express his thoughts in a very strong way. Some might even call it aggressive, but that is not the point here. What is more concerning, is that this means a lot of security measures won\u0026rsquo;t find its way into the kernel. This includes those who might make the operating system more secure, or easier to secure.\nConclusion Security is hard, that has been proven over and over again. Linux security is no different. Sure, there are a lot resources available, but they can be overwhelming and confusing. One of the reasons includes the contradicting advice. Other reasons are: unclear risks for each software package, and sometimes limited understanding of security concepts, by both users and developers. Even with all this complexity, I hope Lynis provides you better insights. Let\u0026rsquo;s stop complexity together! We will definitely keep fighting the endless battle to make Linux security easier.\n","permalink":"https://linux-audit.com/linux-security-fails-to-be-simple/","tags":["linux","security"],"title":"How Linux Security Fails to be Simple"},{"categories":["Network"],"contents":"DNS Configuration on Linux We often don\u0026rsquo;t realize the importance of DNS, or name resolving in our infrastructure. The impact when things go (slightly) wrong is huge. Time to have a good look at improving our DNS configuration.\nHow DNS resolving works When your Linux system needs to know the IP address of a particular host, it will use gethostbyname(3) function. This will use the nsswitch configuration stored in /etc/nsswitch.conf. For the related hosts line, it will determine how to do resolving.\nThe order specified, determines how resolving will tried for each lookup\nIt is common to find the usage of the word hosts, which refers to the /etc/hosts file, a static list to be configured by the system administrator. It is then often followed by the word dns, which specifies it can use DNS queries to get the answer. Here our journey begins to query nameservers. To know which nameservers should be used, the /etc/resolv.conf file is consulted. Each nameserver is prepended with the word nameserver, followed by the IPv4 or IPV6 address of a DNS resolver.\nUsers who use systemd, might actually have a \u0026ldquo;resolve\u0026rdquo; in their nsswitch.conf configuration, pointing to the systemd-resolved service. It performs cache and aggregation of DNS related settings.\nTimeouts and settings Most Linux administrators, have a minimal configuration stored in the /etc/resolv.conf file. To counter the impact of an unreachable DNS system, we can do a few things.\nDefine multiple nameservers The easiest option is adding more nameservers. If you have defined just one, you are fully relying on the availability of that particular system. Also when that particular nameserver has a higher load of queries to process, your services will be affected.\nNote: Most Linux distributions use /etc/resolv.conf directly. Some use /etc/resolvconf.conf , or /etc/systemd/resolved.conf when using systemd-resolvd. So check carefully what your system is using before making adjustments.\nLimit DNS timeout In the resolver configuration, we can change the timeout of DNS queries. If we don\u0026rsquo;t get an answer within a specified amount of time, we continue using the next system. By default, this can be as long as 30 seconds! That means if a nameserver is not available, it will take a while before it tries the next one. And typically it won\u0026rsquo;t remind that this nameserver is down, so other DNS requests may have to experience the timeout as well. So change your configuration file and set it to a much lower time.\noptions timeout:1\nMaximum DNS resolve attempts Typically it doesn\u0026rsquo;t make sense to repeat the same request multiple times to the same system. This is especially true when the related system is down. Restrict the number of attempts:\noptions attempts:3\nDivide requests It doesn\u0026rsquo;t always make sense if all systems use the same system. The name resolver allows you to rotate the requests. So the first request might go to 10.0.0.1, while the second goes to 10.0.0.2.\noptions rotate\nThere is a little caveat: rotation can have a negative impact. The performance of a busy nameserver is typically better, as it can serve data from its cache. So the busier the nameserver, the quicker it usually is to respond to requests. Therefore use this setting only when you are sure that all involved nameservers are fairly busy.\nLocal caching for Speed The quickest network packet is the one which does not need to travel the network. For most systems a lot of repetition is involved in the tasks it is doing. This also applies for DNS requests. Very often a request will be made for a host we recently already contacted. Linux systems do not cache DNS requests by default. This means that a lot of traffic is sent to the network, for nothing!\nWe can counter the repetition of requests towards our nameservers, by using caching. These tools can run locally, and cache both positive and negative matches. In other words, it has a different timer for both types. If a name or IP address can\u0026rsquo;t be found, it may be caching that result shorter (or longer). This improves the caching table and hit rates on the cache, while limiting the bad impact of a \u0026ldquo;miss\u0026rdquo;.\ndnsmasq One of the options to consider is the toolkit dnsmasq . It can provide DHCP and DNS services for smaller networks. It is also a great candidate to run just a local DNS cacher. After installation, configure it so that it just does DNS resolving. Then provide it with some nameservers, so it can externally request the answers. Last step is pointing your normal resolver configuration to 127.0.0.1, so dnsmasq will be the middle man to deal with DNS requests.\nRelated packages Arch Linux - pacman -Ss dnsutils Related utilities These utilities might be handy during configuration, testing, and troubleshooting.\ndrill dig host ","permalink":"https://linux-audit.com/networking/linux-dns-tuning-for-performance-and-resilience/","tags":["dns","performance","system tuning"],"title":"Linux DNS Tuning for Performance and Resilience"},{"categories":["FreeBSD","Identity and Access Management"],"contents":"Linux and *BSD systems have by default a root user installed. As it has a user ID of zero (0), it gains the highest level of permissions from the kernel. On FreeBSD systems, there is also the \u0026rsquo;toor\u0026rsquo; user, with the equal high-level user ID of zero. It is simply the reversed version of \u0026lsquo;root\u0026rsquo;, and installed as a backup account. By default, it has no shell assigned, so it can\u0026rsquo;t log in.\nWhy keep the toor user? Some BSD users strongly suggest keeping the toor user, as it can be used during system recovery. Others actually use this user instead of root and apply hardening to the root user, so that is only can be used on the console.\nTip: be careful with using bash or other shells on your high-privilege users. If the upgrade of such shell fails, you might be locked out. For that reason, it might be good to keep it at the default C shell.\nIs there a good reason to remove it? If you don\u0026rsquo;t use the toor user at all, simply remove it. A healthy security mantra says: everything unused should preferably be removed from the system. This helps to reduce the so-called attack surface of a system. Use vipw to edit your password file and remove the related entry.\nConclusion The toor user is a piece of history on FreeBSD systems. Some people like it, others think it is unneeded. If you don\u0026rsquo;t use it, simply remove it.\n","permalink":"https://linux-audit.com/what-is-the-toor-user-on-freebsd/","tags":["authentication"],"title":"What is the ‘toor’ user on FreeBSD?"},{"categories":["Vulnerabilities"],"contents":"What is Linux privilege escalation? Privilege escalation is the process of elevating your permission level, by switching from one user to another one and gain more privileges. For example, a normal user on Linux can become root or get the same permissions as root. This can be authorized usage, with the use of the su or sudo command. It can also be unauthorized, for example when an attacker leverages a software bug. Especially this last category of privilege escalations is interesting to understand, so we can better defend our Linux systems.\nHow to escalate privileges? Attackers who try to obtain additional privileges, often use so-called exploits. Exploits are pieces of code with the goal to release a particular payload. The payload will focus on a known weakness in the operating system or running software components. This may result in the software crashing or giving access to unexpected areas of the memory. By overwriting segments of memory and executing special crafted (shell) code, one may gain a successful privilege escalation.\nThese are the steps an attacker usually takes:\nFind a vulnerability Create the related exploit Use the exploit on a system Check if it successfully exploits the system Gain additional privileges It is all about enumeration The first step is to find a weakness or vulnerability in the system. To learn about any weaknesses you have to know what operating system and version is used. This is done with a process that is called enumeration. Within this process, you try to learn as much as possible about a network and its systems.\nAttackers find more information by using Google, port scanning, and study the responses of requests from applications. With each step, more information becomes available. A similar approach is taken by penetration testers (pentesters), attackers with a legal contract to do so.\nDuring this enumeration phase, the attacker can also determine if there are any compilers are available. If not, then there might any high-level programming languages like Perl or Python instead. This information is useful for a later stage, in which exploit code is used.\nAs part of enumeration, a lot of data will be collected. Every finding has to be stored, so it can be stored and processed later. Each piece of information can be used to search for known vulnerabilities, or other entries into the system. For example, when Apache is used, and the version number is listed, we can search for known vulnerabilities for that particular version.\nLinux enumeration For most operating systems and applications there are dedicated tools to help. Linux enumeration tools focus specifically on retrieving data from several key areas. These include directories that store the system configuration or its status, like /etc and /proc. There are several system administration tools available that will retrieve network details, file locations, or the system version. Example for such files include:\n/etc /proc Relevant commands:\nip lsof ss uname Exploiting the weakness Next stage is about exploiting any weaknesses found. Sometimes ready-to-use code can be executed against the target, resulting in some level of access. Your WordPress installation (or a plugin) might be outdated, which may give an external visitor the permissions to upload files. The attacker can use this to plant a custom PHP script, to collect more information from the system. This is done by using specific PHP functions, like system(), to execute commands on the system itself.\nHow an attacker exploits software The exploit process may take different steps before the right level of access is gained. Just being able to upload a file might be harmless to the system. So with every step, the attacker tries to retrieve more information and adjusting any required exploit. Sometimes a vulnerability might be there, but not exploitable. This can be due to additional defense layers (e.g. memory randomizing). The attacker has to adapt to the specifics of the machine.\nExample of a privilege escalation attack To show how an attacker may become root, let\u0026rsquo;s have a look at an example. Let\u0026rsquo;s assume the following: we have a Linux system running CentOS, with Apache and a WordPress website on it. Like most WordPress installations, it has several plugins installed. The webmaster had a busy period and did not update the plugins for a while. This is how a privilege escalation attack could go:\nThe attacker runs an automatic script to detect this outdated plugin on many systems across the internet The automated script picks up on the presence of the plugin on the system and checks if it is version 1.2.4 The attacker verifies the finding (or weed out any false positive) Attacker manually abuses the weakness in the plugin and via that uploads a custom PHP file to the system The attacker now requests to run this PHP script, to retrieve more data on the system The output of the script finds the availability of a compiler The script also finds an outdated Linux kernel, which has a known exploit to become root for non-privileged users A small C program is uploaded via the plugin The compiler is executed to compile the specific piece of C code into a binary program The program is executed to abuse the Linux privilege escalation bug in the kernel A new user is added to the system by the attacker The attacker can now log in to the system via SSH This is just an example of how a small piece of information is used during enumeration and followed up for later processing. Then the process is repeated several times to find more details about the system until the attacker gains full root permissions.\nHow do you defend against privilege escalation? The best way to counter Linux privilege escalations is by using the common \u0026ldquo;defense in depth\u0026rdquo; method. You apply several defenses, each targeting a specific area. If one layer of defense fails, this doesn\u0026rsquo;t necessarily mean your system can be compromised. That is obviously easier said than done, so let\u0026rsquo;s have a look in some of the measures.\nReduce the information leaked by applications Most applications have an application banner. This can be a greeting message with details about the application, like its name and version number. While it may look innocent, it is better to avoid giving away too much information. Especially leaking version numbers should be prevented.\nHiding the nginx version number WordPress hardening and reduce information disclosure Remove compilers or restrict access to them The presence of a compiler is not needed for most systems. Production systems should only have a compiler available when it is absolutely necessary. As attackers often need the compiler to successfully build an exploit, removing them is definitely a good step.\nApply Linux updates and patches Systems often get compromised due to weaknesses in software components. There are actually multiple suggestions in this area. First of all, subscribe to mailing lists to know what kind of vulnerabilities were found recently. Next step is to run updates on a regular basis and keep your systems up-to-date. Also, apply security updates automatically when possible, like using unattended-upgrades on Debian and Ubuntu systems.\nRun file integrity monitoring software The best way to detect a privilege escalation or breach is by monitoring important system files. If one of them change unexpectedly, this may be an indication of a security issue. This monitoring can be achieved by file integrity monitoring (FIM) solution. Popular tools include AIDE or with the Linux audit framework (auditd).\nPerform system auditing Maybe the best thing one can do is running continuously security audits. For Linux systems, consider a tool like rkhunter or ClamAV to do malware scanning. Use Lynis for an in-depth security scan of the system. While Lynis is intended as a defensive tool, it actually can find things that are related to privilege escalation. Think of issues like cronjobs that are writable or showing software banners. For that reason, Lynis is also used by pentesters in their work. System auditing may actually reveal unexpected vulnerabilities that the usual vulnerability scanners could not find.\nPrivilege escalation checkers Some tools can help you with checking if there is a privilege escalation possible. This can be a useful exercise to learn how privilege escalations work. They will also help you check if your Linux systems are vulnerable to a particular type of privilege escalation and take counter-measures.\nunix-privesc-check - Gather information and determine possible attacks LinEnum - Perform enumeration and check for possible Linux privilege escalation options Have a look at the privilege escalation tools on Linux Security Expert for more options and more extensive reviews.\nConclusion Linux privilege escalation can happen due to one or more failing security layers. An attacker has to start doing enumeration and process the resulting data. He or she will continue to do testing when more information becomes available. This will repeat until one of the security defenses gets penetrated. Applying proper security defenses is your first safeguard against these attacks. They get much stronger if all defenses are in place, like minimizing the data you share, applying security updates, and monitoring the systems.\nDid you learn something from this article? Great! You can actually contribute by sharing this article or by sending in feedback. Thanks!\n","permalink":"https://linux-audit.com/understanding-linux-privilege-escalation-and-defending-against-it/","tags":["clamav","compiler","ifconfig","linux","pentest","penetration testing","privilege escalation","vulnerabilities"],"title":"Understanding Linux privilege escalation and defending against it"},{"categories":["System Administration","Time"],"contents":"The network time protocol helps computer systems to synchronize their time. We know this protocol by its shorter name NTP. In the past, it was not really a big issue if your system was a few minutes off. This changed with the interconnected world we are now living in. One of the better examples is networks relying on the authentication protocol Kerberos. If your system time is not correct, you may not be able to authenticate. This is because granted tickets have a built-in protection against timing attacks. While you may not be an attacker, the system will refuse to work when it finds requests being from the past or future.\nWhen your local clock is not correct, serious damage could happen. Database data and log files could be incorrect, resulting in data loss at worst. For forensics, it might become very hard to reconstruct the steps occurred in a security incident. So having your Linux systems happily synchronized is a must. Let\u0026rsquo;s have a look how things work and how we can troubleshoot when things don\u0026rsquo;t work.\nHistory of Time We relied in the past on the system itself, to maintain a time. This was done by using a hardware component, which is named the real-time clock (RTC) . But no device or component is 100% reliable, so your system time could slowly become \u0026ldquo;outdated\u0026rdquo;. If it went a little bit too quickly, you would be living in the future, according to your computer. For other systems, they would be living in the past. Systems are nowadays connected to other networks. This makes it possible to synchronize our times to very precise clocks. We call those atomic clocks . Instead of using digital components, they use the radiation of atomic particles. Then we can share the time with radio waves, so other systems can get synchronized.\nLinux and Time Most Linux systems use the following options to synchronize time\nNo synchronization NTP daemon NTP client Other clients No Synchronization The first option \u0026ldquo;none\u0026rdquo; is obvious: there is no software installed on the system to maintain the time. While this may sound as a guarantee of getting out of sync, it isn\u0026rsquo;t always the case. Virtualized systems for example, may use the host system to get the right time. When starting such a system, they get the right time of the host, and be able to maintain it correctly during uptime. There is a risk of \u0026ldquo;skewing\u0026rdquo; (getting out of sync) if the client system is not able to count the cycles correctly, e.g. when the CPU speed is adjusted. Another risk is when the host system does not always give each client the same amount of time per CPU cycle, resulting in small variants in counting.\nNTP Daemon Next option is a NTP daemon. For Linux is typically a running process, or daemon, with the name ntpd. This process is waiting to receive time from several trusted sources. When it knows with a certain guarantee what the time is, it will instruct the kernel to use this new time, and synchronize it usually also with the hardware clock. This way hardware clock, Linux kernel and NTP daemon have the same understanding of the time. When the NTP daemon sees some skewing again, it will adjust the time again.\nThe process of time adjusting usually happens in small steps. This way other software on the systems doesn\u0026rsquo;t suddenly get confused. For example: it is now 4:43:52 PM and we would log something to a file. Then our NTP daemon decides to change the time 10 minutes back in time. Three minutes later we log another line to our file, which will be suddenly 4:36:52 PM. Not only does this get confusing in log files, it may corrupt data in databases and processes relying on network synchronization.\nCommon daemons\nntpd openntpd (OpenBSD project) NTP Client A much simpler option is using a NTP client. It does a similar thing as a NTP daemon, except that it does not track the time from many sources. Instead, it requests the time of a trusted source, and acts upon that information. A tool like ntpdate or rdate are used this way, and scheduled by a cron job to regularly check the time and synchronize.\nCommon clients\nntpdate rdate Other Clients The last category is the other clients. When using virtualized systems this option might be used. Usually there are guest tools available, which is installed on the client system itself. It will do system householding in the background and exchange data with the host system, such as the time.\nTime troubles As with most software, things can go wrong. Many of us rarely check if our time sources are properly configured and still work correctly. We just assume the time is correct and the system does the synchronization correctly, right? Especially when using a NTP daemon, things can go wrong. Its configuration needs to be set-up correctly, and checked regularly. If not, sooner or later, time will skew and result in being a few minutes off.\nFalsetickers The first category of NTP troubles is when using a so-called falseticker. Like our own system can be incorrect, a trusted time source can be incorrect. It can be happening on purpose, misconfiguration, or hardware issues. If we rely on such a resource, our time will be wrong as well. If you are using the NTP daemon together with ntpq, these false-tickers can be recognized with a \u0026ldquo;x\u0026rdquo; in front of the entry.\nStratum 16 Another thing to check for is the \u0026ldquo;stratum 16\u0026rdquo; entries. We refer to an atomic clock or a reference clock as stratum 0. Stratum 1 devices collect the time from a stratum 0 device, usually via radio waves (GPS, CMDA, etc). Then our own systems are usually at stratum 2 or 3. If an entry shows stratum 16, something is wrong. It might not be able to synchronize its date. This may be occurring when it can\u0026rsquo;t find the source. Something as simple as iptables filtering too much traffic.\nUnreliable Sources The last category consists of sources which are unreliable. Because the NTP daemon receives time information from a configured set of systems, it will check them with regular intervals. It will compare the data received from the sources, and take factors like distance and network delay in account. When it finds that a source provides unexpected results, it will be marked as unreliable. You can solve this by using different sources which are closer to you, or even internal. If it already an internal network source, then something might be wrong with the device. Most likely multiple systems will mark the same system as unreliable. When using a NTP daemon (and ntpq), these items are marked with a minus (-).\nTime out of sync Good to know is that NTP daemons usually won\u0026rsquo;t synchronize in big steps, as previously described. If time is too far off, it may even stop functioning, which is on purpose. This is an indirect warning that the time should be correctly manually. Best way to handle this is stopping first all process relying on time synchronization. Then manually synchronize time with a tool like ntpdate or rdate.\nDiscover Time Issues So now we know it is important to track the time, and keep it synchronized it properly. Using the ntpq utility we can query the details of our time synchronization. In particular, we can see what sources are used, and any issues.\nNo sources can be reached, showing stratum 16\nThe best way to discover time synchronization issues is by monitoring the output of ntpq when using a NTP daemon. If you are using a NTP client, then it would make sense to compare it to trusted source and see if it does not differs too much (e.g. a few seconds). You could add tests to your monitoring tool to validate your time configuration on a regular basis.\nFor those who already use our security auditing tool Lynis, you are covered when using a NTP daemon. Lynis will parse the output and inform you if any false-tickers, or unreliable sources are used on your Linux system.\n","permalink":"https://linux-audit.com/troubleshooting-linux-time-synchronization-with-ntp/","tags":["clock","ntp","time","time synchronization"],"title":"Troubleshooting Linux Time Synchronization with NTP"},{"categories":["Lynis"],"contents":"Lessons learned between our last and current release\nThe Lynis project team is proud to announce a new release of our security auditing tool . With months of work and a variety of changes, we bumped up the version to a \u0026ldquo;zero release\u0026rdquo; (2.2.0). The technical changelog is included in the download. We consider it to be a stable release, yet ask all to test it first.\nBeing the original author of Lynis, there is an additional background behind a changelog, which might be even more interesting. With this post, I want to share some of the background going into open source development. We have both our challenges and victories. Let me share some of our insights, in the same \u0026ldquo;open\u0026rdquo; spirit we develop our software.\nAchievement: 2nd Place at ToolsWatch One of our biggest lessons to share directly, is that it doesn\u0026rsquo;t really matter how good software you create, or at what price. People simply expect that the quality is good, and the price is as low as possible. It is hard for most open source projects to be cheap and provide high-quality software without a budget. Instead of focusing on those aspects alone, get noticed and extend your community. We applied this principle, resulting in a second place at the yearly ToolsWatch vote for the best security tools . A nice achievement and another public reference that people trust Lynis. This trust will help the project for the next years, like sharing it on conferences, blogging about it, or sharing it with colleagues and friends.\nLong (Overdue) Release In the last years, we released Lynis very regularly. That is, varying from every few weeks, to a week of eight. This time, it is almost 6 months between the previous release and this one. We reflected on that, to better understand why it took much more time than usual. In essence, it comes down to priorities.\nIn 2013 we decided to create a commercial offering around Lynis, to ensure the open source tool received a continuous feed in resources (time, expertise, development, promotion). In its turn, the open source version is also the client component in Lynis Enterprise. A better Lynis client benefits both the community and our customers.\nOpen source should provide value, additional services should provide additional value\nBut customers don\u0026rsquo;t pay for something they can get for free. That is the reason why we invest a lot of our time into the development of Enterprise functionality. Happy customers enable us to continue the work on all components. In other words, we set our priorities straight, to ensure long-time success.\nThere are serious consequences of not releasing an open source project often. One is that people might think it is no longer developed or supported, and then abandon it. They might even speak bad about the project, stating it is useless. We have learned that lesson before, so we know how big the impact can be. It is one of the reasons why we made the jump to GitHub, to show the community things are being developed and continuously updated.\nRelease open source project often, for community display, damage control, and marketing purposes\nCrazy as it might sound, but open source projects needs a similar marketing approach as commercial software. Even if you give away your software for free, its value is low if it is not being used. The more people using your project, the better it becomes over the years. More eyeballs, as they say in marketing.\nDue to the long time between releases, we have collected a lot of changes. Several new tests have been added, and even more enhancements than before. To prevent the release coming too big without testing, we did intermediate releases on GitHub. This way the community still has the latest and greatest, with the possibility to test and improve.\nCommunity Speaking about community, we have the same challenge as many open source projects. It is hard to find long-time contributors. Most have busy jobs and can join in quickly for some improvements, before they are filled with other activities. This is a big lesson for us, and we are doing now the ground work to make it easier to do small commits. Those who do contribute for a longer time are rewarded by a special mentioning in our changelog.\nOur first priority after this release is getting our documentation improved. Before we had a single very long document, with tried to capture all the important things. If you want people to use your open source project, give them the chance to quickly try it out.\nIf you have an open source project, your documentation should provide an outcome\nWe provide now our \u0026ldquo;Getting Started\u0026rdquo; document to target beginning users. Initial review of our analytics showed that it works. Enough reason to split up more documents and turn them into in expected outcomes. Before one clicks on a document, it should be clear what they can expect and what result they will achieve.\nSimplicity Sells During the development of this release we learned that a few things were still to complicated. Our software does not have to be installed, nor has dependencies. You would think people would be able to figure out quickly how to use it, right? We were wrong and decided to improve upon that as well.\nWe already used colored output in previous versions. It does not sound spectacular as a feature. Still, our guess is that not many shells scripts have it implemented properly. This release we applied coloring on more places, including the first screen people get when running the tool without parameters.\nColored output to guide first-time users\nSmall changes like this help simplifying the usage of the tool, especially if you never used it before. Another common issue is the overload of options used. In this release we stripped actually some of the parameters from the output, and put them only in the man page and online documentation.\nAdding new features is great, removing clutter is even better\nNew users are easy confused. Some don\u0026rsquo;t speak the English language, others stare at all the available options. So our focus is now on reducing things on the screen. They can only confuse new users, or they are screen clutter for the typical user. We can\u0026rsquo;t track parameter usage, otherwise it would have been a great source to determine what other parameters should be left out. What we can do is better divide the use cases of our software. In other words, we are trying to learn better how people really use our software. Not just the environment they are running in, but the how, why, and when.\nUsers will do unintended things with your software, if you give them room for misinterpretation\nWe are now interested in how they run the tool, which version, and what they do with the results. We do ask people now, after seeing a recent GitHub request was about the usage of a particular parameter, then parsing the output of the screen. This was not intended, as we actually have a report file with the findings, so you don\u0026rsquo;t have to scrape screen output. It made us realize that users will do unintended things with your software, if you give them room for misinterpretation. Another lesson is that we should bring it in line. So if on screen something is flagged as a red item, it should be in the report as well.\nWhy Upgrade? As this is the non-technical changelog, we won\u0026rsquo;t go into the technical changes. It would still be useful to determine the reasons to upgrade. One of the most interesting facts we learned during this development cycle is that people upgrade for different reasons. We already learned that some people actually refused to upgrade, simply because the packaged version in their Linux repository was outdated. So this is a clear reason why some do not upgrade. Now let\u0026rsquo;s dive into the reasons we heard why people do actually want to upgrade.\nSome of our users wanted to use the latest code, to be sure they were not missing out on features and bug fixes. Running the latest version does not scare them, knowing they can review changes first before pushing them to all systems. Especially those who embedded the GitHub repo in their deployment belong to this category. Some were running the latest code simply because the different in version number.\nDon\u0026rsquo;t confuse users with version numbering of your open source project\nWe had our previous release tagged 2.1.1, and after a batch of commits we updated the release number on GitHub, without making it into an official release. We used it to determine if people were up-to-date when providing bugs. While great for us, it was confusing for our users. After all, they found version 2.1.8 on GitHub, but still the \u0026ldquo;outdated\u0026rdquo; version 2.1.1 on our website. So don\u0026rsquo;t confuse users with version numbering of your open source project. It will only result in more support questions.\nThe Need for Packages For a long time, we resisted the idea of creating Lynis packages. Yes, packages are a good thing and help with system management. At the same time, our tool didn\u0026rsquo;t need any installation and rolling your own package is not that hard. Last year we continuously heard that people want things packaged. Although we can\u0026rsquo;t offer it yet, we started with some initial work to get things in motion on this front as well.\nMore Non-Technical Changelogs? If you enjoyed this article and found it useful, share it with your (online) friends. I personally appreciate a comment on how it may help your project, and count it as a +1 for doing it more often.\nThanks for being part of the open source community.\nMichael\nRunning Linux, Mac OS X or some other UNIX-based system? Don\u0026rsquo;t guess your security levels, but check it easily yourself. Within 2 minutes you have your system audited with the free and open source tool Lynis. And if you like it, consider the upgrade to the Enterprise version.\n","permalink":"https://linux-audit.com/non-technical-changelog-insights-of-6-months-development/","tags":["documentation","lynis","open source","software development"],"title":"The Non-Technical Changelog: Insights of 6 Months Development"},{"categories":["Software"],"contents":"The unattended-upgrade tool is a great way to keep your system automatically updated. While you might not always want to do that for all packages, it definitely can be a great way to assist in your security efforts. In that case, tell it to track security updates and install the related packages.\nIf you are using third-party packages (e.g. via PPAs ), the system has no idea about security updates for those packages. So you need to take an additional step and get them included manually.\nDetermine the PPA Origin and Suite The first goal is to determine the details from the PPA (or other external package type). This can be done by peeking in the /var/lib/apt/lists directory. Use the related files ending with InRelease, to see more details about the specific package.\nless /var/lib/apt/lists/ppa.launchpad.net_nginx_development_ubuntu_dists_trusty_InRelease\nFor our nginx package we get this output below.\nThe two things we need from this file is the field Origin and Suite. These two strings have to be combined and provided to unattended-upgrade. It then understands that this PPA should be upgraded automatically.\nChange Configuration File vi /etc/apt/apt.conf.d/50unattended-upgrades\nIn this case, we add nginx to the Unattended-Upgrade::Allowed-Origins section.\n\u0026ldquo;LP-PPA-nginx-development:trusty\u0026rdquo;;\nSo the result will look something like this:\nThe hardest part has been completed!\nPerform Automatic Upgrade When the changes have been made, check the new configuration. Run unattended-upgrade in dry-run mode. Add the debug flag to see more details.\nunattended-upgrade -dry-run -debug\nIf there is an update available, which you can check with apt-get upgrade, then it should show up. If not, your might have a typo or mismatch in your repository name.\nAdditional Tips Sometimes it is good to run a development version, especially if you need the feature set. Keep in mind that those packages are not part of the security channel. So additional upgrade attention for these packages is strongly advised. Better safe than sorry!\nThe unattended-upgrade tooling can\u0026rsquo;t always upgrade packages. This is especially the case when configuration files are changed. And you guessed it right, this happens a lot to development packages. So if you have the chance, set up additional monitoring for any upgrades. Don\u0026rsquo;t simply rely on the existence of unattended-upgrade, and have a second tool or script test the availability of updates.\n","permalink":"https://linux-audit.com/upgrading-external-packages-with-unattended-upgrade/","tags":["software management","software patching","unattended-upgrades"],"title":"Upgrading External Packages with unattended-upgrade"},{"categories":["Software"],"contents":"The world has changed a lot in the last era, especially when it comes to computing. This applies also to the services we run on our Linux systems. Some of these services (like rlogin), were previously the defacto tools to do administration. Now they are considered to be bad and insecure.\nWhat makes a service insecure? Services can become insecure when they have characteristics like:\nNo (or weak) authentication No (or weak) encryption Insecure protocols Running as root Authentication insecurities One example might be if a program only requires a password or pin, without any information like an username. This happens often in physical solutions, but sometimes also in software. The risk involved is two-folded: it is easy to guess and provides no (or weak) accounting.\nWeak encryption Some programs come without any form of encryption. This doesn\u0026rsquo;t mean all programs are insecure by default. It depends on the task they have. When communicating via the network, it rapidly becomes a risk, as data can be sniffed. If any confidential data, like authentication details, is shared over the link, the service is \u0026ldquo;leaking\u0026rdquo; this information.\nAnother risk with encryption is when weak protocols and ciphers are used, or its implementation is flawed. A good example here is the usage of SSL version 3, which was previously fine. Now it is considered to be weak (e.g. POODLE attack).\nInsecure protocols Protocols define basic rules on communication and exchange of information. Some of the older protocols are now considered to be weak. This includes protocols which lack encryption, yet provide the possibility to authenticate. The credentials by the user can be easily sniffed on the (inter)network\nExamples:\nFTP HTTP IMAP NIS POP3 SNMP v1/v2c Where possible avoid using these protocols for services which share sensitive data.Running as root\nRunning as root Most software has the risk of being exploited at some stage. To prevent the impact as much as possible, processes should not run with root privileges. Doing so might give away the full system when one layer of your security defenses is breached. Fortunately, most well-known Linux daemons drop privileges.\nConclusion Implement proper authentication methods for your software. If you are dealing with sensitive information, make use of encryption and ensure the right ciphers and protocols are being used. Run your software with the least amount of privileges, and enable logging of important, or security-related events.\n","permalink":"https://linux-audit.com/find-disable-insecure-services-linux/","tags":["authentication","cryptography","development","login","software"],"title":"Find and Disable Insecure Services on Linux"},{"categories":["System Administration"],"contents":"Mosh, or mobile shell, is the ideal tool for remote system administration. While SSH is great, Mosh beats it in several areas. Let\u0026rsquo;s dive into the reasons why it makes sense to learn about Mosh.\nPros Session resumption Remember the last time your connection was interrupted? It it frustrating and sometimes even leads to losing some of your work. The stable TCP connection is not always a blessing. Mosh comes to the rescue, especially for less stable connections. It solves this issue by picking up where you left. Mosh has a roaming function, allowing you to even between connections. Very useful when you are on the move, or your WiFi connection provides you suddenly with a new IP lease. No longer you need to run everything in a screen session.\nNo root permissions needed Mosh can run without root privileges. This is because it uses normal binaries (mosh, mosh-client, and mosh-server). There is no daemon (of its own) waiting for incoming connections.\nDefault UTF8 support Every terminal reacts differently to \u0026ldquo;strange\u0026rdquo; characters. Mosh will not break your terminal, as it uses UTF-8 by default. So the intended output ends up correctly on your screen, every time. This is much better than showing garbled text or even hanging your terminal screen.\nResponsive SSH has the tendency to be slow to respond to your Ctrl+C requests. This is caused by network buffers be filled and your Ctrl+C has to wait in a long line. Mosh can deal with this, and ensures you it quits much quicker. Interestingly enough Telnet was in some ways much better than SSH, like local echo. Mosh brings back some of the good features.\nAnother great use-case is when having to do administration on slow connections, especially with \u0026ldquo;long\u0026rdquo; network links, including a high latency). With SSH you are waiting for every character to show up, Mosh makes it much more responsive. It does so with the combination of previous input and predictions. It shows what it expects to be there, by using underlining. Then it does a validation step to ensure things are right and tells you that by removing the underlining.\nHow Mosh works Mosh uses SSH to do authentication. So instead of reinventing the wheel, it leverages the available basic components of the system. After it performed the authentication via SSH, it will fire up a server component (mosh-server), which runs as a process by the user itself. The client will then use connect to this mosh-server process via a new channel. This channel uses UDP (opposed to TCP for SSH) and consists of a UDP port in the higher port range (60000-61000). Everything is encrypted, using AES-128 in OCB mode .\nFirewall rules One of the disadvantages of Mosh is that the additional UDP port means opening up a set of ports in your firewall. As one port per connection is used, you can limit this (e.g. 60000-60005). For environments which strict rules, this might be a deal breaker. Still for many situations Mosh is a useful addition to simplify work.\nIPv6 support Mosh had a disadvantage in the networking options, which was the lacking IPv6 support. Fortunately, that is solved and initial IPv6 support is now available.\nInstallation Convinced Mosh can help you in your work? Great, then it is time to do the installation. Mosh works on pretty much all UNIX-based systems, like Linux, Mac OS X, *BSD, Android, and even in Chrome.\nHere are some quick starters:\nArch Linux - pacman -S mosh CentOS and Fedora - yum install mosh Debian and Ubuntu - apt-get install mosh or use the PPA for mosh-dev More details can be found on the project page , or on GitHub .\nClients for Android JuiceSSH supports Mosh as well.\nSo how is your experience with Mosh? Love to hear it!\n","permalink":"https://linux-audit.com/mosh-ssh-alternative-for-system-administration/","tags":["ssh","system administration"],"title":"Mosh, the SSH Alternative Option for System Administration"},{"categories":["Web"],"contents":"Key pinning can be tricky and sometimes you might encounter a website having an incorrect key pin. This is usually caused by renewing certificates. In that case the duration time of the key pin might overlap the expire time of the moment of renewal.\nChrome Error You will be seeing an error something like:\nYour connection is not private Attackers might be trying to steal your information from domain.com (for example, passwords, messages, or credit cards). NET::ERR_SSL_PINNED_KEY_NOT_IN_CERT_CHAIN What Does It Mean? HSTS , forces websites to always use HTTPS. This is done by instructions of the related web server. An additional layer is using HPKP HTTP Public Key Pinning. This allows web masters to define what public keys related to the SSL certificate are good. The client then stores these for a specified amount of time.\nSometimes things go wrong with this key pinning, resulting in a website to be unreachable. In that case, you can delete the related key pin manually. This does not apply if the key pins of the domain are preloaded into the browser (e.g. of Facebook). In that case a browser update is needed.\nDeleting a HSTS key pin Fortunately solving this error is simple, by removing the domain from the HSTS database. T\nGo to the URL chrome://net-internals/#hsts Now delete the related domain After deleting the domain, the related key pins will be removed as well. You should be able to visit the related website again.\n","permalink":"https://linux-audit.com/web/delete-a-hsts-key-pin-in-chrome/","tags":["hpkp","hsts","web","web browser"],"title":"Delete a HSTS Key Pin in Chrome"},{"categories":["System Administration"],"contents":"The Dandified YUM tool, DNF, has become a powerful package manager for systems running Fedora. As it looks now, it will become also the default package manager for CentOS 8 and RHEL 8. One of the benefits from dnf is the option to retrieve security information very easily. This allows us to use it for automatic security patching of our Linux systems. Let\u0026rsquo;s explore the options and see how dnf-automatic can help us with fully automated patching.\nSecurity Patches The newer versions of Fedora use DNF. To check available security patches, use the dnf command.\ndnf updateinfo list security\nDNF found security patches for glibc vulnerability\nWhile this output is helpful, we want more automation, right? Instead of creating our own shell script for the cause, we leverage the dnf-automatic utility to do the heavy lifting for us.\nInstall and Configure dnf-automatic We can deploy updates automatically with the package dnf-automatic. This package simplifies automatic patching, by running on a timer and then apply updates. You can configure it to just install security updates.\nNote: For those who previously used YUM, this is similar to the \u0026ldquo;yum -update security\u0026rdquo; command, but better.\nFirst step to using dnf-automatic, is to install the package.\ndnf install dnf-automatic\nThe common output of command installation should show up.\nInstallation of dnf-automatic\nConfiguration of automatic updates Next step to apply updates security updates only, is to adjust this /etc/dnf/automatic.conf. Configure at least the following settings:\napply_updates = yes\ndownload_updates = yes\nupgrade_type = security\nScheduled timers After applying the changes, you are done with the configuration part. Now check the status of the related timer, to see if that is activated.\nsystemctl status dnf-automatic.timer\nThis timer will be disabled by default. If this applies to your system as well, enable the timer and start it.\nsystemctl enable --now dnf-automatic.timer\nThat looks much better. Time for the last steps and validate that everything works as expected.\nTesting dnf-automatic After the configuration, let\u0026rsquo;s test it. This way we know we made the right changes to the file. Run dnf-automatic manually to check if it runs properly.\ndnf-automatic\nIf you have any security patches available, you will see some output on the screen. When none updates are available, the output will remain empty.\nThat feels like a much safer system again\nUnfortunately, the logging of dnf-automatic is limited. You can view the installed updates in /var/log/dnf.rpm.log file. This will show what packages are upgraded and installed.\nJan 26 10:01:30 INFO Upgraded: openssh-7.1p2-1.fc23.x86_64 Jan 26 10:01:30 INFO Upgraded: bind-license-32:9.10.3-10.P3.fc23.noarch Jan 26 10:01:30 INFO Upgraded: bind-libs-lite-32:9.10.3-10.P3.fc23.x86_64 Jan 26 10:01:30 INFO Upgraded: bind-libs-32:9.10.3-10.P3.fc23.x86_64 Jan 26 10:01:33 INFO Installed: kernel-core-4.3.3-301.fc23.x86_64 Jan 26 10:01:38 INFO Installed: kernel-modules-4.3.3-301.fc23.x86_64 Jan 26 10:01:38 INFO Upgraded: dhcp-libs-12:4.3.3-8.P1.fc23.x86_64 Jan 26 10:01:38 INFO Upgraded: dhcp-common-12:4.3.3-8.P1.fc23.noarch Jan 26 10:01:38 INFO Upgraded: bind99-license-9.9.8-2.P3.fc23.noarch Jan 26 10:01:39 INFO Upgraded: bind99-libs-9.9.8-2.P3.fc23.x86_64 Jan 26 10:01:39 INFO Upgraded: dhcp-client-12:4.3.3-8.P1.fc23.x86_64 Jan 26 10:01:39 INFO Installed: kernel-4.3.3-301.fc23.x86_64 Jan 26 10:01:39 INFO Upgraded: bind-utils-32:9.10.3-10.P3.fc23.x86_64 Jan 26 10:01:40 INFO Upgraded: openssh-server-7.1p2-1.fc23.x86_64 Jan 26 10:01:41 INFO Upgraded: openssh-clients-7.1p2-1.fc23.x86_64 Jan 26 10:01:41 INFO Upgraded: rsync-3.1.1-8.fc23.x86_64 Jan 26 10:01:41 INFO Upgraded: perl-PathTools-3.62-1.fc23.x86_64 Jan 26 10:01:41 INFO Upgraded: libnghttp2-1.6.0-1.fc23.x86_64 Jan 26 10:01:41 INFO Upgraded: chrony-2.1.1-2.fc23.x86_64 Jan 26 10:01:41 INFO Cleanup: bind-utils-32:9.10.3-7.P2.fc23.x86_64 Jan 26 10:01:42 INFO Cleanup: dhcp-client-12:4.3.3-7.fc23.x86_64 Jan 26 10:01:42 INFO Cleanup: bind-libs-lite-32:9.10.3-7.P2.fc23.x86_64 Jan 26 10:01:42 INFO Cleanup: bind-libs-32:9.10.3-7.P2.fc23.x86_64 Jan 26 10:01:42 INFO Cleanup: bind99-libs-9.9.8-1.P2.fc23.x86_64 Jan 26 10:01:42 INFO Cleanup: openssh-clients-7.1p1-6.fc23.x86_64 Jan 26 10:01:42 INFO Cleanup: openssh-server-7.1p1-6.fc23.x86_64 Jan 26 10:01:43 INFO Cleanup: bind99-license-9.9.8-1.P2.fc23.noarch Jan 26 10:01:43 INFO Cleanup: bind-license-32:9.10.3-7.P2.fc23.noarch Jan 26 10:01:43 INFO Cleanup: dhcp-common-12:4.3.3-7.fc23.noarch Jan 26 10:01:43 INFO Cleanup: dhcp-libs-12:4.3.3-7.fc23.x86_64 Jan 26 10:01:43 INFO Cleanup: openssh-7.1p1-6.fc23.x86_64 Jan 26 10:01:43 INFO Cleanup: rsync-3.1.1-7.fc23.x86_64 Jan 26 10:01:44 INFO Cleanup: perl-PathTools-3.60-1.fc23.x86_64 Jan 26 10:01:44 INFO Cleanup: libnghttp2-1.3.3-1.fc23.x86_64 Jan 26 10:01:44 INFO Cleanup: chrony-2.1.1-1.fc23.x86_64 Additional steps With software and security patching it happens that related processes need to be restarted. DNF has a plugin command available to test this.\ndnf needs-restarting\nIf nothing has to be done, the output will be empty. Otherwise, it will show you the processes which need a restart. Depending on what has been patched, that can be a long list. Sometimes a fresh system reboot is easier (and safer).\nOutput of dnf needs-restarting\nMeasure after patching While applying security patches automatically has a lot of pros, ensure that your system monitoring is in place. Also, configure remote logging on your systems by using a remote syslog server. This gives you an audit trail of the packages installed and any changes to them. When something breaks, you know at least what happened.\nHappy patching!\n","permalink":"https://linux-audit.com/automatic-security-updates-with-dnf/","tags":["dnf","fedora","linux","software patching","software vulnerabilities"],"title":"Automatic Security Updates with DNF"},{"categories":["Malware"],"contents":"Malicious software plague computers for more than 40 years. It is hard to think this threat will ever stop. The Linux platform definitely has their share of malware, although many people never experienced it firsthand. Let’s dive into this subject and discover why your system might actually being compromised at this very moment.\nThe types of malware To understand the risks, you have to understand the threats and weaknesses. When we talk about malware, there are different family types, each with their own threat and method of attack. The most common five families are:\nVirus - attaches itself to binaries (e.g. ELF binary) Worm - spreads via the network, e-mail, file transfers Rootkit - alters the system with a specific purpose Backdoor or Trojan horse - allows for secret access Dropper - disguises itself as legitimate, while performing secret actions Each of these malware families has a specific usage. Some are meant to spread itself to as many systems as possible. Others are focusing on the opposite and cover their tracks and remain silent. That is, until the master shows up with a secret token.\nViruses on Linux? A computer virus has the intention to attach itself to another file in the first place. Next step is getting distribution. This is very similar to seeds that are hidden inside fruits, to be eaten by animals. The seeds are tough and only leave the animal at a different spot, including fresh fertilizer. When we talk about computer viruses the same thing happens. The file gets infected, moved or copied by a computer user, then deployed on a different system.\nMS-DOS and Windows were known to be very vulnerable for computer viruses. Software was pirated and infected EXE files found their way to new computer systems. As Linux consists of mostly free and open source software components, the chance for a traditional virus are much lower. Why copy a file from someone, while you can download it yourself for free? There are some viruses available to infect ELF files, the typical binary format for Linux binaries. The effect of ELF infecting viruses is minimal, as the chance of spreading is low.\nWhat about Linux worms? Like a virus, a worm wants to get replicated. Instead of being passive and attaching to a binary, it has more aggressive techniques. It becomes active and finds way to spread quickly and perform a related action. To spread it often uses network connectivity, e-mail, or access to file shares. Just being deployed on many systems is not enough. The worm has a second purpose, to do something directly, or after a predefined set of time. This could be harmful, like destroying the hard disk of the infected system.\nWhile most worms were written in the past for Windows, this threat is applicable to Linux systems as well. The Adore and Slapper worms are examples which infected unpatched Linux systems very quickly. This was done by exploiting weaknesses in network-based services and then deploy the worm on the new system. This newly infected system would start scanning other possible targets as well.\nRootkits are a Linux thing As the name describes, rootkits sound like they are applicable to Linux. We can safely say that this is the case. Rootkits are a set of utilities (=kit), to maintain high privilege access (=root). The goal is not to become root, but stay on the system for as long as possible. This way attackers can use the resources of the system at a later stage, without being detected in the meantime.\nTo achieve invisibility, rootkits take a lot of measures to hide itself. This starts with masking any related files or directories on the disk. To achieve this, a binary like /bin/ls or /usr/bin/find needs to be altered. Any legitimate system administrator will no longer see the files when scanning through the system. Then there is the alteration of tools like ps and lsmod, to prevent showing the related processes and kernel modules. Quickly this becomes a set of altered binaries, which is exactly why we call it a kit. Most of the rootkits also have a backdoor in them. This way the attacker can get a free pass to access the system at any time. Without any suspicious logging entries in the log files of course.\nWhere is the backdoor on Linux? Backdoors are common on all platforms, including Linux. They are part of rootkits, but can also be implemented as small standalone pieces. For example, the PHP/C99Shell is one of the most used backdoors on PHP. It simply disguises itself somewhere on the system, often as a file with the JPG or GIF extension.\nThe problem with backdoors is that they can reside on many places within the system. A kernel module could be providing one, only to open up if the right sequence of network packets was noticed. Your SSH daemon could also be backdoored, giving access to those with the master password. So this threat of backdoored services on Linux is something we should take seriously.\nDroppers The dropper is an interesting malware family. It disguises itself as a normal file. Upon execution it does the behavior of something expected, like showing a program. In the background it does a little bit more, often completely unnoticeable for the system user. Droppers exist also for Linux, but are less common. One of the most common one is Linux.RST.B. It infects an ELF file (like a virus), with the goal to provide a backdoor in the background.\nHow do attackers get in? With all these types of malware, you might wonder how you a system gets infected in the first place. There are two main reasons:\nVulnerabilities in software Weak configuration Software products need continuous attention. It is often created under pressure and chances are high it contains 1 or more issues, or software bugs. Some of these bugs may result in a security vulnerability. Such a vulnerability could be skipping a required authentication step, or filling the memory with unexpected data. An attacker can use vulnerabilities to exploit the software, and do things which were not intended.\nWeak configurations are also responsible for many break-ins. This could be the usage of a weak or default password. With so-called brute-forcing these passwords can be quickly detected and abused. In the same category, there are the weak software configurations itself. Your Linux system may be running Apache with all default settings enabled. This instance of Apache may be giving away too much details about the software, but also the operating system itself.\nAttackers use all the tricks in their book to find weaknesses. They start with the enumeration phase, the act of collecting information. Every bit of information is then stored and later analyzed to determine if systems have any possible weaknesses. The more information your systems share, the easier it becomes to find a vulnerability sooner or later.\nWhy is there malware on Linux systems? Most malware on Linux has the goal to do something right now, or at a later stage. From being a bot in a huge botnet, to providing anonymous internet access for the attacker. A lot of the malware is used for sending e-mail spam, or become part in a distributed denial of service (DDoS). Every system with a connection to the internet is therefore a valuable resource to attackers.\nMaximizing malware value Attackers want to get the most value out of their malware. If you would like to misuse a system for sending spam, you don’t want it to be detected too easily. Attackers know this and try to keep their victims systems under the radar as well. Another way to stay undetected is the tricks used by most rootkits and many of the backdoors. Simply disguise as another process, hide somewhere between the thousands of files, or become an invisible kernel module.\nMalware creators have become very creative in hiding their traces. Here are some common places:\nAs a binary in common PATH Somewhere in the directories of the man pages Hidden in a cron job Malware Detection All these mentioned tricks make it close to impossible to detection, so is there hope?\nStep 1: Detect Linux malware locally The first action to take is run specific malware scanners on your Linux system. Don’t do it just after you feel a system may be compromised. Make it part of ongoing scanning and be prepared.\nTools Chkrootkit ClamAV Linux Malware Detect (LMD) Rootkit Hunter Chkrootkit and Rootkit Hunter are tools which focus on Linux rootkits and backdoors. ClamAV and Linux Malware Detect focus more on backdoors (e.g. PHP) and generic malware, including samples used on Windows. Combining them will give you the most chance to detect any trace of malware.\nStep 2: Detect Linux malware by change Another good way to detect malware is using integrity tools. For example monitoring your most critical files for change. If your /bin/ls binary was changed, but you didn’t make any updates to it, that is a bad sign.\nTools AIDE Samhain Step 3: Detect Linux malware on the network Sometimes the best way to detect something is wrong, is outside the system itself. The network is a great place to monitor. See an increased in SMTP traffic from a system, but can’t find anything in the mail.log on the system itself? It might be the result of a hidden process being a member of a botnet and recently woken up. Early detection might save a lot of time and abuse of resources.\nWhat to do after you found something? Dealing with malware can be a challenging task. After all, you have to know first how serious the intrusion is. Your ClamAV discovering an infected email is something completely different than a rootkit on the system.\nDon’t unplug the system if you want to do analysis. Better is to keep the system running in a controlled way, so it can’t do harm. Some malware is smart enough to detect when it is being analyzed, or when it feels it is being busted soon. Such trigger could be a network link going down. Instead, keep the link up, and place the system in a different network segment where it can’t do harm. If you manage your network infrastructure, it would be wise to have a quarantine VLAN available for infected systems. Here systems still have connectivity, but limited. Some traffic might be silently dropped, to ensure the system doesn’t do damage.\nEasiest and still the best advice: if you had a serious malware piece on your system which needed root permissions to install, reinstall your system. If someone else became root, there is always the chance that something is running in the background.\nBut I want to clean up my system! If you feel confident that your system will be really clean after your actions, go for it. That is if you really know how the malware arrived on the system. Next requirement is that if no root permissions were needed to place the malware. Otherwise you may risk maintaining a system with still hidden services running on it. In that case, it may be better to learn a lesson from it: install a fresh system, restore data, implement the right countermeasures.\nTip: Be aware that malicious traces could be in your backup. So only restore data directories and reconfigure the rest manually.\nIs prevention possible? The best way to prevent malware is by applying the right security defenses. We call this process system hardening. Limiting incoming connections on systems can be of great help and avoids exposing vulnerable services. Those services which need to be available can be hardened by using safe configurations. Remove default accounts and change default passwords.\nLinux system hardening takes some time, but is worth the effort. Since this is a broad subject and covers a lot of areas, we have written many individual posts. If you are completely new to the subject or want to understand how to prioritize system hardening, use Lynis. It gives a first good insight what you can do. Start with the warnings, then move on to the quick wins on the list. Linux system hardening is not a one-time effort. It needs to be applied in small steps. So track your progress and prioritize.\nAnother way to prevent malware on Linux is by keeping software up-to-date. This process is named software patch management and consists of receiving security bulletins, testing available patches, and finally applying them.\nAdditional Resources Did you like this article? Here are more resources to follow-up on the subject.\nPresentation - Handling compromised Linux systems Intrusion detection: Linux rootkits Monitoring Linux Systems for Rootkits Got questions to keep your systems protected? Feel free to ask!\n","permalink":"https://linux-audit.com/malware/dealing-with-linux-malware-insights-by-the-author-of-rkhunter/","tags":["backdoor","linux","linux security","malware","rootkit","virus"],"title":"Dealing with Linux Malware, Insights by the Author of rkhunter"},{"categories":["Authentication"],"contents":"Password files on Linux are used to store user details, like your unique user ID and name. It defines who you are on the system, your access to files, and the permissions you have. Proper password management, together with file integrity of your password files, is important. It keeps your system and user accounts safe.\nPassword Files For most Linux distributions there are two related files available: /etc/passwd and /etc/shadow . The first file defines what local users are available on the system. Fields include an identifier code (your user ID), your username, and an informational field named GECOS . This last field provides (optional) details about you, like contact details. The /etc/shadow file is a cloned file of /etc/passwd, with the exception that it does store the related password of users. It has different file permissions, to avoid others from snooping at your password, or a hashed form of it.\nWhy File Integrity Testing? One of the overlooked areas on Linux systems, is actually the integrity of your password files. After all, when everything works, why bother looking into it? One reason is that small errors there, might result in serious security errors at a different place. The password files are an important defense mechanism to allow only authorized people and processes to have access to system resources.\nOne of the things that can happen is a double user ID. This might be by accident, or on purpose (intruder). The best example is two users with a user ID of zero. Normally you just want the root user to have this ID, and definitely not any other use.\nWhat Can be Wrong? With every file we edit, a lot can go wrong within the file itself. As we are dealing here with a password file, duplication and typos are the most common. Let\u0026rsquo;s have a look at the things we have to check for sure:\nFields The easiest check to do is counting the amount of columns. It should cover all specified fields, from UID to the GECOS field.\nUnique IDs Every user ID should be unique. While it was common in the past to give multiple system administrators the UID of zero, this is considered now bad practice. The usage of sudo is well known, and should be used instead. The same principles apply for group files /etc/group and /etc/gshadow . The user ID itself should be within the range of 0-65535, or more restricted by what your /etc/login.defs specifies.\nHarmony with Shadow Files The shadow file /etc/shadow should have the same lines as the files they are connected to. So if you have a user in /etc/passwd , it should be in the shadow file as well.\nUsernames To avoid issues with usernames, they should only use normal characters. The maximum length on Linux is 32 characters. Otherwise, the username will be considered invalid by tools like useradd. Also it may contain numbers, but not strictly of numbers.\nMissing Home Directories For a normal user it would be strange to have an account, yet no home directory. It might drop the user into the root directory, directly after logging in.\nChanging Your Password Files the Right Way To prevent most of the integrity issues, changes should be made with the vipw or vigr command.\nvipw\nThis command is a wrapper utility. It makes a temporary copy of the file, then performs integrity tests before writing it to (final) password file.\nIf you want to change your group files, or shadow file, then use -g (groups), or -s (shadow). Combine them both for changing the shadow group file (/etc/gshadow). Or if you feel rather using vigr, use that instead.\nVerification of Password Files with pwck We love automation. Fortunately, most of the checks above can be done quickly with the pwck utility. The first time you run it, we suggest doing a read-only check.\npwck -r\nThe output might look something like:\nNow it is time to check its findings and determine the impact of each finding.\nFound some issues in your password files or got questions? Let it know!\n","permalink":"https://linux-audit.com/authentication/file-integrity-of-password-files/","tags":["file integrity","password","pwck","system integrity"],"title":"File Integrity of Password Files"},{"categories":["Web"],"contents":"HPKP is a great technology to pin a certificate to a website. On first use of a domain, the browser of the client checks if key pinning is available. Upon a next visit, the browser applies an additional check if the certificate(s) provided is available in the previous list of white-listed sites.\nHPKP error Sometimes things go wrong with HPKP and you won\u0026rsquo;t be able to access a particular page.\nThe best action is first to contact the website and see if they are aware of the problem. Then if you feel comfortable with \u0026ldquo;overriding\u0026rdquo; the error message in Firefox, continue reading.\nDelete the key pin manually Close your web browser first. Then open the file SiteSecurityServiceState.txt in your profile directory. For example:\n~/.mozilla/firefox/aabbccdd.default/SiteSecurityServiceState.txt\nFor Mac users: /Users/[username]/Library/Application Support/Firefox/Profiles/[random].default\nDelete the related domain entries. Save the file and start your browser again.\nFull Error Below the full text for your convenience.\nAn error occurred during a connection to _domain_. The server uses key pinning (HPKP) but no trusted certificate chain could be constructed that matches the pinset. Key pinning violations cannot be overridden. (Error code: mozilla_pkix_error_key_pinning_failure) * The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. * Please contact the website owners to inform them of this problem. Did this page help? Great! If not, let us know, so this article can be improved!\n","permalink":"https://linux-audit.com/web/deleting-outdated-hpkp-key-pins-in-firefox/","tags":["certificates","hpkp","ssl","web browser"],"title":"Deleting Outdated HPKP Key Pins in Firefox"},{"categories":["Software"],"contents":"Checking Security Updates for your Software Packages DNF is the default package manager since Fedora 22. As it is considered to be a better version of YUM, some of our Lynis users asked for DNF support. With focus on auditing and security patching, we definitely wanted to see that for ourselves. While building support, I\u0026rsquo;ve gathered the most important commands. In this blog post we will have a look how we can leverage the DNF output to show only the available security updates.\nCheck for Updates The first step is to check if there are updates. With the check-update parameter we tell DNF to refresh its database and determine if there are any updates available.\ndnf check-update\nShowing DNF Security Updates Next step is to determine if there are security notices available. This can be done with the updateinfo parameter.\ndnf updateinfo\nThis command will tell us how many security updates it has found. Unfortunately, it does not reveal the specific packages involved. To get this information, we need to do a more specific query. By adding \u0026ldquo;list sec\u0026rdquo;, we can tell it to list only the security related updates.\ndnf -q updateinfo list sec\nYour output might look something like:\nDNF in action\nIn this case, the Fedora server found 10 security notices. The number of packages might be higher, as some packages are related to each other (e.g. several instances for bind, dhcp, kernel, openssh).\nThe beauty of DNF is that you can run it as a non-privileged user. So if you are building automated tests, no need to use root permissions.\nMore Automation! If you want to perform an in-depth security scan, you might be happy to know that initial Lynis patches have been applied to support DNF. It will gather installed packages, vulnerable packages, and perform an integrity check of the package database.\nAny of the commands (not) working for you? Let it know!\n","permalink":"https://linux-audit.com/showing-available-security-updates-with-dnf/","tags":["dnf","fedora","package manager","packages","security updates","software management","software patching","software vulnerabilities"],"title":"Showing Available Security Updates with DNF"},{"categories":["File Integrity","Passwords"],"contents":"The password files are an important cornerstone of the security of your Linux system. Commonly they are /etc/passwd and /etc/shadow . Sometimes we receive questions what the right permissions of these files should be. Therefore this blog post to have a look at the file permissions (and ownership) of both files.\nPasswd file The password file stores local accounts of the system. It is a readable text file and uses colons (:) to separate the fields. In this file the account names, identifiers, and other descriptive fields are stored. This file helps with converting user IDs to names (and back).\nExample output of a /etc/passwd file on Linux\nFun fact: some systems which have this file broken (or their authentication like LDAP), will get something like\nI have no name! You don\u0026rsquo;t want to end up being a number ;-)\nPermissions of /etc/passwd While it is fine that all users can read this file, they should not be able to change fields. Otherwise it could disrupt file permissions and authorizations. It would be fairly easy to take over the root account for example.\n# ls -l /etc/passwd -rw-r--r-- 1 root root 1046 Oct 27 16:05 /etc/passwd This file typically has no other special file permissions, like an immutable bit.\nShadow file Like the passwd file, the /etc/shadow file inherits most of the same fields and values:\nExample of /etc/shadow and encrypted passwords\nOne big exception with the passwd file, is the password itself. So in other words, the password is not stored in /etc/passwd, but in /etc/shadow. It is stored as a long string of characters, which is a combination of the hashing algorithm, optional salt applied, and the hashed password itself. If you are new to the subject, then consider a hash like a fingerprint of the password, but not the real content. So you can always check again the rightful owner of the password, without storing it unencrypted. The salt value adds more randomness to the mix. This forces attackers to use a brute force attack on a much bigger set of possible values.\nPermissions of /etc/shadow The owner of the /etc/shadow file is usually the user root. The group is often set to an administrative group, like shadow. Other users are not allowed to read the file directly, to prevent them from gathering hashes passwords of others. With a tool like passwd, which has a setUID bit, the file can be altered in a controlled way.\n# ls -l /etc/shadow -rw------- 1 root root 823 Dec 7 19:59 /etc/shadow Issues Commonly when people have made changes to the files, issues arise. Your password files could be damaged, or have altered file permissions. If that is the case, compare the permissions with another system and correct them. If you didn\u0026rsquo;t make changes to the system, consider that your system may have been compromised by an intruder. In that case, we suggest further analysis and a reinstall of the system.\nDid this article help you fixing your file permissions? Let it know!\n","permalink":"https://linux-audit.com/file-permissions-of-the-etc-shadow-password-file/","tags":["etc","file permissions","passwd","shadow"],"title":"File permissions of the /etc/shadow password file"},{"categories":["Firewall","Network"],"contents":"The seasoned Linux administrator will be familiar with iptables , the network traffic filter. If you ever configured a Linux system with an ethernet bridge configuration, you might even have worked with ebtables. Or possibly you wanted to filter ARP traffic and used arptables? Newcomer nftables has arrived, with the purpose to replace iptables, ip6tables, ebtables and arptables. As with every big upcoming change, it is good to know the differences. We explain what makes nftables different to iptables, and why you want to adopt it in the near future.\niptables VS nftables Simplicity in syntax The biggest change you might like is the simplicity. With iptables, we have to configure every single rule and use the syntax which can be compared with normal commands. So we run iptables with -A INPUT -s 192.168.1.20 etc. With nftables, we have a much simpler syntax, which looks like BPF (Berkeley Packet Filter). The syntax of nftables is inspired on the tcpdump syntax. This means shorter lines and less repetition.\nExample:\nnft add rule inet traffic-filter input tcp dport { 22, 80, 443 } accept\nCombined rules The example above includes another big improvement: combined rules. So instead of repeating lines for every single port, we can combine them. This is useful for UDP/TCP ports, and also ICMP types.\nExamples:\nConfigure IPv6 table and input chain\nnft add table ip6 traffic-filter nft add chain ip6 traffic-filter input\nAllow several IPv6 ICMP packets\nnft add rule ip6 traffic-filter input icmpv6 type { nd-neighbor-solicit, echo-request, nd-router-advert, nd-neighbor-advert } accept\nMultiple actions One rule can hold multiple actions. With iptables, this would mean splitting rules, and jumping to different blocks.\nnft add rule ip filter input ip protocol vmap { tcp : jump tcp-chain, udp : jump udp-chain, icmp : jump icmp-chain }\nProtocols combined Like the option to combine multiple actions, nftables allows defining one rule that will support both IPv4 and IPv6. Much better than using iptables and ip6tables and synchronizing rules between the two.\nBuilt-in support for sets To use lists or sets with iptables, you need to install ipset. Nftables has integrated set support and it can be used more naturally within the configuration.\nConcatenated value pairs Within sets and maps, fields can be combined for further evaluation. For example the combination of an IP address with a port number. Instead of making individual rules, this data can be put into data array and then later used.\nnft add element traffic-filter dict { 192.168.0.1 : drop, 192.168.0.2 : accept }\nMore flexibility With iptables you have several default base chains. With nftables you always start with a blank slate. Just add what you need, from chains to rules.\nImproved performance One of the important changes is that nftables is optimized for speed. This is achieved by using data structures, which help with quick lookups in memory. These data structures can directly be used within rules. For example, you can tell in your rule that you want to use a particular field, like an IPv4 address and take a particular action (verdict). This way nftables knows how to handle the fields and apply quicker lookups on them.\nIntelligence and protocol support The userland utility nft holds the intelligence on what is supported and passes it to the kernel. This means that when a new protocol needs to be supported, you don\u0026rsquo;t have to rebuild your kernel. Instead, extending the nft utility will in most cases be sufficient.\nEasy data export For those wanting to store the configuration, there is an export option available. Nftables supports exporting in XML and JSON output.\nnft export json\nMonitoring and logging Optional counters If you need log counters, nftables allows you to set them on-demand. They are optional, to keep overhead at a minimum.\nnft add rule inet traffic-filter input tcp dport ssh counter accept\nLive tracing support Troubleshooting rules in iptables is not that easy. The nftables developers created a tracing option that can be set on a rule. After it has been set, run the nft monitor trace command. You can optionally add one or more -n flags for more details.\nnft -nn monitor trace\nConclusion The features of nftables and its usage, look very promising. Especially its simplicity brings it more in line with the way pf works on BSD systems.\nDid you find any other major difference which was not mentioned? Share it in the comments.\n","permalink":"https://linux-audit.com/networking/nftables/differences-between-iptables-and-nftables-explained/","tags":["bpf","firewall","ip6tables","ipset","iptables","linux","linux security","network","nft","nftables"],"title":"Differences between iptables and nftables explained"},{"categories":["Network"],"contents":"Version 6 of Internet Protocol is now 20+ years available. You would think it is widely available now, right? Not exactly. Still many internet providers don\u0026rsquo;t have it deployed for their customers. Hosting companies are not always eager to deploy it either. Mostly because of lacking knowledge. To get at east more knowledge shared on the security side of IPv6, we have crafted this guide. Hopefully it will be a practical guide for your to configure and tune your configurations.\nThe Need for a New Version The addressing scheme for version 4 was thought to sufficient at the time. With the widespread usage of the web and services on the internet in general, this turned out differently. With some trickery, we were able to extend the possibilities, like subnetting and running multiple websites on one single IP address. Last year, the last pools of available IP addresses were depleted and it is time to move forward.\nVersion 6 of the Internet Protocol promises that address space will never be a problem anymore. Even if we were to give every human, device, and trees, their own address. We will see if that holds true in the next 20 years to come.\nDo I Really Need It? Most guides on the internet tell you a simple thing: disable IPv6 if you don\u0026rsquo;t need it. That sounds logical and is a good hardening principle. Unfortunately, this advice might also be harmful. For example, the web really benefits if we make the transition to IPv6. Instead of fearing away for some of the known attacks, it is better to build up a good understanding of the subject and take the right measures. Still, if you feel IPv4 will do in your internal network, then use that and disable IPv6. In any case, keep reading and make the world a safer place.\nBefore we start talking about hardening, it makes sense to do a gentle introduction into IPv6. After all, there are still a lot of people who never used or configured it.\nIPv6 Basics Although the subject is very extensive, with lots of standards and RFCs, here are the basics of the new protocol.\nNo ARP (neighbor discovery instead, based on ICMPv6) No broadcasts (lots of multicast instead) Hierarchical address space (with clear functions and separation) Multiple IP addresses per interface, with different scopes Header is simplified, modular due to extensions and chaining Neighbor Discovery Instead of using ARP, multicast is used other to detect other systems and routers. For the discovery of routers there are Router Advertisements and Router Solicitations, for other systems Neighbor Advertisement and Neighbor Solicitation. These advertisements and solicitations help hosts to find their way on the network. Depending on the configuration, the right device(s) will respond to a query. For example the initial task to obtain an IP address with the right prefix.\nIPv6 Addresses on First Sight If you never worked with IPv6 before, the new addresses can be overwhelming. No longer the short IP addresses we are used to, divided by dots. Instead, we get colons and additional letters. To avoid this blog post becoming a book, we will skip on all the details regarding addressing. Instead, let\u0026rsquo;s have a look at some common addresses:\n::1/128 - Localhost (IPv4 alternative: 127.0.0.1) fc00::/7 - Unique Local Addresses (ULAs) (IPv4 alternative: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) fe80::/10 - Link-Local Addresses, used on the internal network (IPv4 alternative: 169.254.0.0/16) 2001::1000::/32 - Global addresses (Teredo) 2002::/16 - Global addresses (6to4) The other addresses are usually external addresses (global), or special purpose addresses. If you want to know more about these address types, use the RIPE IPv6 sheet as a handy reference.\nLet\u0026rsquo;s leave addressing and move towards configuration.\nIPv6 Configuration Basics Linux distributions are all slightly different. In this guide, we will show several examples. For specific configuration of your favorite Linux distribution, have a look at the More Resources section, located at the end.\nBefore doing any configuration at all, why not first determine what is configured? This commands are also helpful for auditors and security professionals, to understand configurations better. Previously the ifconfig command was used by many distributions, with the ip command being its common replacement. First step is determining what interfaces are configured, and in particular which have a version 6 configuration.\nip -6 addr\nThis command shows address information per interface, highlighting only the interface specifics of the version 6 family. For example, any link-local IPv6 address ends with a suffix: scope link. Interfaces or addresses with a global static configuration, have the suffix scope global static. Systems configured via SLAAC will show scope global dynamic. When private extension address is used, it will show scope global temporary dynamic, and for expired items: scope global temporary deprecated. As you quickly will see on your environments, every system is differently configured. So it is good to know these types exists.\nIf your Linux distribution uses systemd, changes are high that you are using the systemd-networkd service. Querying its status can be done with systemctl.\nsystemctl status systemd-networkd\nThe status output of systemd will show something like \u0026ldquo;Gained IPv6LL\u0026rdquo;, when it received a Link-Local address. As with the ip command, the output depends on the type of address being configured on the interface.\nThese two tools can already provide some insight. Time to get closer to the in-depth configuration of IPv6 itself.\nManual versus Automatic Configuration Time to make changes to the system and set up IPv6. The first choice to make is if we want to use automatic configuration, or do it by hand. The first option simplifies things a lot. Using manual configuration gives you more control regarding the configuration. It also counters a few known attacks, so for specifically hardening purposes it is the best bet. The best option for your environment? It really depends on your networking architecture and what type of machines are in it.\nAutomatic Configuration Two common ways for providing addresses to new and existing clients, is using SLAAC (Stateless address auto-configuration) or DHCPv6. First question you might ask yourself, is which one to use. Unfortunately, the answer depends on your environment, used devices and their support for IPv6.\nA quick dive into both:\nSLAAC Uses Multicast ICMPv6 Provides IPv6 prefixes Router advertisement provide addressing, routing, MTU and other options. Privacy issue: uses MAC address (which can be countered with privacy extensions turned on) DHCPv6 When using the stateful configuration of DHCPv6:\nClient/Server Uses multicast UDP Provides addressing, routing, NTP, SIP, DNS, other options The stateless DHCPv6 configuration will use SLAAC with the flag O=1 (Other Config) set, followed by extra configuration options via DHCPv6.\nSystem Configuration on Linux Systems Time to apply some configuration!\nArch Linux When using netctl, use the following line and add it to your interface configuration file:\nIP6=stateless\nIf you are using NetworkManager, you should receive automatically an IPv6 address, when advertised. When using the newer systemd-networkd, reload with:systemctl restart systemd-networkd\nsystemctl restart systemd-networkd\nCentOS Change the related network configuration file, with the right interface. For example /etc/sysconfig/network-scripts/ifcfg-eth0.\nIPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no Debian and Ubuntu Change the /etc/network/interfaces file.\niface eth0 inet6 auto\nNote: newer versions of Debian and Ubuntu most likely don\u0026rsquo;t use this file anymore\nManual Configuration If you don\u0026rsquo;t want to use SLAAC or DHCPv6, it is time to make some serious changes. First step is turning of auto-configuration. Add the following lines to your /etc/sysctl.conf\nnet.ipv6.conf.all.autoconf=0\nSystems running Red Hat distribution like RHEL, might need another change in their network configuration file, which is usually /etc/sysconfig/network.\nNETWORKING_IPV6=yes\nIPV6_AUTOCONF=no\nOn a regular basis people are advised to also disable the router advertisements (RA). This is done with the sysctl key net.ipv6.conf.all.accept_ra sysctl. This is actually not advised. As a starter, these RAs help hosts determining if a system is the same subnet, in other words if they are on-link or off-link. Also the configuration of DNS and packet size (MTU) is arranged with RAs. For neighbor discovery (the IPv6 equivalent of ARP), timeouts, and other settings are determined. So in other words, if you decide to disable RA, make sure you really know what you are doing. You might find trouble later on, which was simply caused by ignoring RAs.\nHere are some tips for the most common Linux distributions to get you started. Note that some of these configurations can change over time, as systemd is gaining in popularity.\nArch Linux and Fedora 21+ Systems using systemd, need to be configured via systemd-networkd. Change file /etc/systemd/network/50-static.network.\n[Match] Name=eth0 [Network] Address=192.168.1.2/24 Gateway=192.169.1.1 Address=fe80::2/64 Gateway=fe80::1 Reload network configuration:\nsystemctl restart systemd-networkd\nIPV6 Configuration for Debian and Ubuntu Change your /etc/network/interfaces file\niface eth0 inet6 static address aaaa:bbbb::abcd:abcd netmask 64 mtu 1280 gateway aaaa:bbbb::abcd::1 CentOS and Red Hat (RHEL) Change the related interface in the directory /etc/sysconfig/network-scripts, starting with ifcfg-*.\nIPV6INIT=yes IPV6ADDR=fe80::100/64 IPV6_DEFAULTGW=fe80::1/64 DNS1=fe80::10/64 DNS2=fe80::20/64 Reload configuration (older versions of CentOS/RHEL)\nservice network restart\nReload configuration (CentOS/RHEL)\nnmcli reload nmcli con down \u0026#34;System eth0\u0026#34; nmcli con up \u0026#34;System eth0\u0026#34; Testing Connectivity For most common utilities there is an IPv6 variant available. For ping this is ping6 and provides similar functionality.\nIf you get a \u0026ldquo;connect: Invalid argument\u0026rdquo; error with ping6, you might wonder why that is. Because of the routing and dual stack involved, you need to specify the interface.\nping6 -I enp0s3 fe80::800:27ff:fe00:1\nAn alternative is leaving out the -I parameter and append % (e.g. %eth0) to the address.\nping6 fe80::800:27ff:fe00:1%enp0s3\nAnother tricky item is the usage of Link-Local addresses on Linux, and the related name resolving. During testing, you may receive the error \u0026ldquo;unknown host\u0026rdquo;. Even if you are certain the hostname and IP address are correct. If you encounter this, ping the IPv6 address manually instead. Use a tool like dig or host, to check if the AAAA records are properly configured. These are the equivalent of A records for IPv4 addresses.\nTroubleshooting To find why things don\u0026rsquo;t work as expected, start using the ip command as shown before. This will provide you the interface details. Other useful sources are dmesg for kernel related messages. Check your syslog and systemd logging.\nThis system won\u0026#39;t be automatically configured as no routers provide IPv6 router advertisements\nIPv6-only or Not? Another decision to make is the communication protocol for the machine. If no IPv4 mappings are needed, a possibility is to go to IPv6-only. Change this particular sysctl key and add it to /etc/sysctl.conf to persist after a system reboot.\nnet.ipv6.bindv6only=1\n**Relevant RFC: **RFC 3493 - Basic Socket Interface Extensions for IPv6\nIPv6 Hardening on Linux Besides the specific configuration type, we can apply additional layers of system hardening to our IPv6 network configuration.\nAs discussed before, disabling router advertisements can be good for hardening. At the same time, it may result in a lot more tweaking. If you decided to go for it, then add the following lines to your /etc/sysctl.conf file.\nnet.ipv6.conf.all.accept_ra=0 net.ipv6.conf.default.accept_ra=0 Enable IPv6 Privacy Extensions When clients receive an address through SLAAC, it is created by combining an advertised prefix with the (physical) MAC address of the network card. If you rather don\u0026rsquo;t create addresses based upon MAC addresses, enable the privacy extensions in IPv6. Add the following lines to /etc/sysctl.conf.\n# Enable the IPv6 Privacy Extensions: do not use MAC address net.ipv6.conf.all.use_tempaddr=2 net.ipv6.conf.default.use_tempaddr=2 Configuration with NetworkManager When using NetworkManager, ensure that it also is configured properly.\n/etc/NetworkManager/NetworkManager.conf\n[connection]\nipv6.ip6-privacy=2\nConfiguration with systemd-networkd For systemd-networkd, open up your configuration file (e.g. /etc/systemd/network/wired.network) and locate the [Network] section. Define IPv6PrivacyExtensions and set to true.\n[Network]\nDHCP=yes\nIPv6PrivacyExtensions=true\nRate Limiting To protect against denial of service (DoS) attacks, limiting resources is usually a good counter measure. At the same time caution is advised, as sometimes changes can actually increase the chance of an accidental denial of service by legitimate usage. So in all cases check the business purpose of your machine and what a typical network load is expected.\nnet.ipv6.icmp.ratelimit - limit in time (ms), default 1000 At this moment, IPv4 has more sysctl keys available than IPv6. The main reason for this is simply that not all features are ported yet.\nnet.ipv4.icmp_echo_ignore_all = 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_errors_use_inbound_ifaddr = 0 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.icmp_msgs_burst = 50 net.ipv4.icmp_msgs_per_sec = 1000 net.ipv4.icmp_ratelimit = 1000 net.ipv4.icmp_ratemask = 6168 Firewalling with IPv6 Firewalls are still a common hardening measure. Its effectiveness to filter out network traffic is high, while rule sets are usually fairly easy to set up. To counter some of the attacks on Linux systems, we can filter out some of the bad traffic involved with known IPv6 attacks.\nip6tables At this moment, ip6tables is still the common firewall to be used. It has proven its stability over the years and helps with filtering out the good and bad traffic.\nnftables Most likely nftables will replace iptables and ip6tables in the upcoming years. It does simplify rule sets a lot. Another benefit of nftables is that the ip and ip6 protocol families can be merged into inet, to further simplify things. No more double rules!\nAllow several types of IPv6 ICMP packets:\nnft add rule ip6 traffic-filter input icmpv6 type { nd-neighbor-solicit, echo-request, nd-router-advert, nd-neighbor-advert } accept What to Filter? Most networks will be using a dual-stack implementation for IP traffic. Therefore it is important to ensure that both IP versions are filtered. It would be a shame if you tightened your firewall, just to find out that via IPv6 everything is available.\nAnother option is to filter out traffic from unexpected ranges. With the earlier mentioned guide from RIPE, you can quickly find out what you want to allow, block the rest of the traffic. Also apply DHCPv6 filtering and restrict access of UDP ports 546 and 547.\nAdditional reads:\nRFC 4890 - Recommendations for Filtering ICMPv6 Messages in Firewalls IPv6 sysctl: Keys and Values Explained With IPv6 enabled on your Linux system, a lot of additional sysctl keys will be listed. Let\u0026rsquo;s break them up and understand each section.\nIPv6 Neighbor Configuration net.ipv6.neigh.[interface].[key] The neighbor configuration settings are applied to an interface. So you the choice to define a particular interface with specific settings. If you want to apply them all interfaces, change your /etc/sysctl.conf and override a setting to the all and default, to make them applicable for all interfaces.\nIPv6 Routing net.ipv6.route.gc_elasticity = 9 net.ipv6.route.gc_interval = 30 net.ipv6.route.gc_min_interval = 0 net.ipv6.route.gc_min_interval_ms = 500 net.ipv6.route.gc_thresh = 1024 net.ipv6.route.gc_timeout = 60 This first batch of keys within routing, determine the garbage collecting. This process cleans out old entries, usually to free up memory and increase performance. If your machine has a very particular role requiring many simultaneous connections or sessions, consider diving into tweaking these. Otherwise, leave them on their default value.\nnet.ipv6.route.max_size = 4096 net.ipv6.route.min_adv_mss = 1220 net.ipv6.route.mtu_expires = 600 The max_size defines the maximum routes in the kernel. Then there is min_adv_mss, the minimum advertised MSS (maximum segment size). It depends on the MTU (maximum transmission unit) of the first hop. Be careful with adjusting this, as it can introduce unexpected network behavior. Last there is the mtu_expires, which defines how long items for the MTU will be cached (in seconds).\nGeneric IPv6 Settings Sysctl: net.ipv6.xfrm6_gc_thresh Garbage collector threshold. Unless you have specific tuning requirements for your machine, keep it at the default value (e.g. 32768).\nFlow labels Flow labels help marking a connection. Normally this is done with the combination of details regarding sender and receiver, port numbers and sequence numbers. Due to fragmentation, encryption and other reasons, it may be hard to know to which previous flow a new incoming packet belongs. Flow labels solve this problem.\nNormally the defaults (1) for both net.ipv6.flowlabel_consistency and net.ipv6.flowlabel_state_ranges will do.\nRelevant RFC: RFC6437 - IPv6 Flow Label Specification\nFirewall Marks If you are using firewall marks, this boolean on net.ipv6.fwmark_reflect will define if the fwmark property will be set to zero, or the value to which fwmark it is applicable. For most users keeping this the default value will be fine.\nICMP Rate Limiting The setting net.ipv6.icmp.ratelimit limits the amount of ICMP packets, expressed in milliseconds. Default value is 1000.\nIPv6 Fragmentation net.ipv6.ip6frag_high_thresh = 4194304 net.ipv6.ip6frag_low_thresh = 3145728 net.ipv6.ip6frag_secret_interval = 600 net.ipv6.ip6frag_time = 60 These keys deal with fragmentation of packets. The time determines how long one can stay in memory, before being thrown out. If you want to apply some hardening, you can lower this, depending on the type of traffic you are serving. The ip6frag_secret_interval setting defines the interval (and lifetime) of a hash secret. This secret is used to ensure fragments can be identified quickly as being from a source we already communicated with. All other unknown fragments might be malicious, with the goal to exhaust system resources.\nCommon IPv6 Actions and Commands Disable IPv6 net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1 Use disable_ipv6 and activate changes with sysctl -p. Change your /etc/sysctl.conf file as well. This way changes remain active after a system reboot. You can also disable IPv6 on a single network interface. In that case define the sysctl setting only for that particular interface.\nOther ways include disabling it in the kernel, like disabling the related kernel modules. Add this to your modprobe configuration, which is usually stored in /etc/modprobe.d/.\ninstall ipv6 /bin/true\nDelete an IP Address ip -6 addr del \u0026lt;address\u0026gt; dev \u0026lt;interface\u0026gt;\nDeleting an address from an interface can be done with the ip command. Note that when auto-configuration is used, addresses may reappear.\nPreference for IPv6 Already using IPv6 within your network? In that case you might prefer AAAA records for systems, instead of A records. You can instruct the resolver to use that instead in your resolver configuration, which is stored in the /etc/resolv.conf file.\noptions inet6\nAs always, test before you deploy.\nPackage Managers and IPv6 Most network services are nowadays aware of the existence of both IPv4 and IPv6. Depending on your configuration, your package manager may need some configuration to work properly as well.\nAPT (Debian / Ubuntu) Use the settings Acquire::ForceIPv4 and Acquire::ForceIPv6 to specify what protocol version you want to use.\nIPv6 Security Testing Utilities There are several packages which focus specifically on testing IPv6 and some known weaknesses. Here is a quick overview if you want to test them in your lab.\nTHC-IPv6 The THC-IPv6 toolkit helps attacking several weaknesses in the IPv6 protocol, including ICMP6.\n6to4test - Check if IPv4 address has dynamic 6to4 tunnel setup alive6 - Detects systems listening to an address denial6 - Denial of Service (DoS) tests againsts a specified target detect-new-ip6 - Detect new IPv6 devices joining the network dos-new-ip6 - Detect new IPv6 devices and causing Denial of Service (DoS) exploit6 - Test for known IPv6 vulnerabilities fake_mipv6 - Take away mobile IP when IPSEC is not used firewall6 - Firewall testing utility flood_advertise6 - Flood utility with neighbor advertisements flood_router6 - Flood utility router advertisements implementation6 - Perform implementation checks parasite6 - Spoofing utility for ICMP neighbor solicitations/advertisements redir6 - Spoof traffic via ICMP6 rsmurf6 - Remote ICMP flood attack utility smurf6 - Local ICMP flood attack utility thcping6 - Custom ICMP packets trace6 - Alternative tool for traceroute6 (with ICMP6 echo request and TCP-SYN) Other great tools:\nsoca nc6 ndisc6 (ICMPv6 Neighbor Discovery tool) Scapy (package manipulation) SI6 Networks IPv6 Toolkit Nmap Evil FOCA More IPv6 Resources There are many resources available regarding IPv6. To help you find more specific details for your platform, we have collected some.\nRelated RFCs IP version 6 has a nice collection of Request For Comments (RFC):\nRFC 1981 - Path MTU discovery for IP version 6 RFC 2460 - Internet protocol, version 6 (IPv6) specification RFC 2461 - Neighbor discovery for IP version 6 (IPv6) RFC 2462 - IPv6 stateless address autoconfiguration RFC 2464 - Transmission of IPv6 packets over Ethernet networks RFC 2465 - Management Information Base for IPv6: Textual Conventions and General Group RFC 2472 - PPPv6 (Red Hat only) RFC 2710 - Multicast Listener Discovery (MLD) RFC 3041 - Privacy Extensions RFC 3056 - Connection of IPv6 Domains via IPv4 Clouds RFC 3315 - Stateful DHCPv6 RFC 3484 - Default Address selection RFC 3493 - Basic Socket Interface Extensions for IPv6 RFC 3596 - DNS Extensions to support IPv6 RFC 3810 - Multicast Listener Discovery Version 2 (MLDv2) RFC 3971 - SEcure Neighbor Discovery (SEND) RFC 4007 - IPv6 Scoped Address Architecture RFC 4193 - Unique Local IPv6 Unicast Addresses RFC 4213 - Transition Mechanisms for IPv6 Host and Routers RFC 4291 - IPv6 Addressing Architecture RFC 4443 - Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification RFC 4890 - Recommendations for Filtering ICMPv6 Messages in Firewalls RFC 5942 - IPv6 Subnet Model: The Relationship between Links and Subnet Prefixes RFC 6104 - Rogue IPv6 Router Advertisement Problem Statement RFC 6105 - IPv6 Router Advertisement Guard RFC 6106 - IPv6 Router Advertisement Options for DNS Configuration RFC 6437 - IPv6 Flow Label Specification RFC 6980 - Security Implications of IPv6 Fragmentation with IPv6 Neighbor Discovery RFC 7381 - Enterprise IPv6 Deployment Guidelines Operating Systems Arch Linux - IPv6 wiki page Gentoo - IPv6 Router Guide Other Hardening guide by ERNW Found this guide helpful, or have other tips to secure IPv6 configurations on Linux? Let it know!\n","permalink":"https://linux-audit.com/networking/linux-security-guide-for-hardening-ipv6/","tags":["hardening","ip6tables","ipv6","netfilter","network","system hardening"],"title":"Linux Security Guide for Hardening IPv6"},{"categories":null,"contents":"Privacy is important, so we limit the amount of information we collect. We stripped most of the external services that we used in the past.\nWhat we still use and why:\nLog file to store and process requests (e.g. abuse of our resources) Webmaster tools (to monitor the health of the website, does not store visitor information) Not in use Technology Rationale Analytics (e.g. Google Analytics) We don\u0026rsquo;t want to store sensitive information if not needed, especially not at an external party Cookies This website does not use cookies to maintain sessions Got any questions regarding privacy and or how we store or use information? Reach out to us via the contact\n","permalink":"https://linux-audit.com/privacy-policy/","tags":["privacy"],"title":"Privacy Policy"},{"categories":["File Integrity"],"contents":"Did you come across a file, but don\u0026rsquo;t know what type it is? Let\u0026rsquo;s learn how to analyze it.\nThe unknown file You may encounter a file on your system with known contents or goal. Usually, the first thing we do is then use cat to show the contents, or execute it. While that makes sense, it may be dangerous to do. It might be a piece of malware, disrupt your screen output or even hang the terminal. Here is a better way to do it, using the file command. Great for forensics, malware analysis, intrusion detection, and normal day-to-day system administration.\nThe file command Most systems will have the file command available. It is a nifty small tool which helps you quickly determine what the purpose of a file is. Besides just telling if it is binary code or data, it will include additional details. For binaries, it may share that it is an ELF binary, for 64 bits systems, how it is linked and if it depends on external function libraries.\nHow does file work? Even veteran administrators might never have looked into the details of the file command, but taken its power for granted. The tool is pretty nifty, because it uses a staged set of tests, working towards a final answer. Depending on the outcome of each test it continues, till it finds useful details to share.\nStage 1: File system tests The file command starts with determining if a file is a \u0026ldquo;simple\u0026rdquo; file. It can be a symbolic link to another file, or a directory. Yes, directories are files as well. To help with this, file uses the stat(2) system call, which is also a standalone utility.\nRegular file is shown by stat utility\nFrom this output, we can see that the stat command does not reveal much. It is considered to be a regular file, which might hold any type of data. So time to go the next phase.\nStage 2: Magic discovery When the file command knows the type of file we are dealing with, it can test more in-depth. This is done via a magic file, which represents many text strings, or character combinations. For example, a file starting with PK might be a compressed file.\nOutput of file -l displaying magic strings\nWith this predefined list of strings and regular expressions, most file types can be discovered.\nStage 3: Text files The last stage is determining if the file is a text file. If it didn\u0026rsquo;t find a match by using tips from the magic dataset, it will assume it is a normal file with text in it. To be sure, it will check the character set used (ASCII, UTF-8). Also if line breaks are used and what type, like applied line feed and carriage returns, which differ between files created in MS-DOS/Windows, Mac OS and Linux systems.\nCommon types of output are:\nASCII text ASCII text, with very long lines gzip compressed data, from Unix, last modified: File Command and Parameters The file utility is very easy to use, as it actually does not require any parameter, except the file you want to analyze. While there are parameters available, most of them cover very specific cases. An example is changing the behavior of the tool, or the output itself.\n--brief - Do not show the file name --uncompress - Uncompress the data file for further inspection See the man page for more specific use cases.\n","permalink":"https://linux-audit.com/how-to-determine-a-file-type-on-linux/","tags":["binary","forensics","how-to","intrusion detection","malware"],"title":"How to see the file type?"},{"categories":["Web"],"contents":"For years, WordPress is used as a platform for blogging. Last years, more and more companies have even built their website in WordPress. Unfortunately, this also means it is more often targeted by scripts, searching for their next victim. The primary reasons for a WordPress hack, are often disclosed information and outdated software components. This is applicable to the WordPress version itself and modules, like the plugins. In this article, we have a look at dealing with unwanted information disclosure, and how we can reduce revealing too much.\nWhat are sensitive details? As a normal user, you often won\u0026rsquo;t notice it: systems leak a lot of sensitive data. Things they should simply not share, like software package names and version numbers. They might look innocent on the first sight. Then if you consider malicious people can use these types of information to do information gather, also known as reconnaissance.\nThe process of information gathering is a phase which both good and bad people use to determine possible vulnerabilities. The more information shared, the easier it becomes to guess or actually know what software is being used. While it is close to impossible to hide you are running WordPress, it is possible to avoid sharing a lot other confidential pieces.\nWordPress shares too much by default Most software packages and plugins come with a readme file to explain the purpose of the software. Often this file also includes installation instructions. With open source projects, there is commonly also a LICENSE file, specifying the related software license (like GPL, MIT, BSD). Then there is the CHANGELOG file, full with details about the changes made. Version numbers are mentioned in the latter, making it is easy to find out what particular WordPress or plugin version is installed. Normally not a problem, if that data was stored only locally. With WordPress however, those files are shared and publicly available.\nWordPress is sharing its version in the HTML code of your blog or website. It might look innocent, but it makes is very easy to find out who is not applying his security updates. With automated scripts searching for this information 24/7, your site is at risk.\nCorrection: deleting generator tag Time to get this piece of HTML deleted on our sites. Unfortunately, the related change has to become part of your theme. This means there is a risk of your changes later being overwritten and you have to repeat it. For now, go to the editor and open up functions.php. This is your theme functions file.Add near the top the following PHP snippet:\nAdd near the top the following PHP snippet:\n// Custom: Remove \u0026lsquo;generator\u0026rsquo; meta tag\nremove_action(\u0026lsquo;wp_head\u0026rsquo;, \u0026lsquo;wp_generator\u0026rsquo;);\nAfter saving, the meta tag \u0026ldquo;generator\u0026rdquo; should be gone. Check by refreshing your page and view the source code. Search for the generator tag and confirm that it is gone.\nCorrection: deleting text files Next step is deleting files which might share program and version information, like README files.\nVia SSH access If you have access to the related system, the find utility will be of great help.\nfind /data/site -iregex \u0026ldquo;.*/(readme|license|changelog)(.md|.txt)?\u0026rdquo; -print\nThis command searches in the /data/site directory. It searching for files with names like readme.txt, but also LICENSE, due to use iregex (instead of regex). Ensure proper escaping, otherwise the find command will not reveal the files.\nWith the command above we also gather files with markdown, as that is now more common to see for documentation. It is similar to normal text files, with additional characters to denote headers and other markup.\nThe output of running find, might be looking something like this:\n/data/site/wp-content/plugins/akismet/LICENSE.txt /data/site/wp-content/plugins/akismet/readme.txt /data/site/wp-content/themes/twentyfifteen/readme.txt /data/site/wp-content/themes/twentyfifteen/genericons/LICENSE.txt /data/site/wp-content/themes/twentyfifteen/genericons/README.md /data/site/wp-content/themes/twentytwelve/readme.txt /data/site/wp-content/themes/twentyfourteen/readme.txt /data/site/wp-content/themes/twentyfourteen/genericons/LICENSE.txt /data/site/wp-content/themes/twentyfourteen/genericons/README.txt /data/site/wp-content/themes/twentythirteen/readme.txt /data/site/wp-content/themes/twentythirteen/genericons/LICENSE.txt /data/site/wp-content/themes/twentythirteen/genericons/README.txt Manually Without access to the system, one option might be to download all files, locally search for *.txt and delete any file which has program related data. The remove them the files from the server and upload from the local copy. Definitely not an ideal option, but it works.\nAnother alternative is checking common directories, like wp-content and wp-includes, and remove the related files by hand.\nPrevention of information disclosure Correction is great, now let\u0026rsquo;s take the next step and try to prevent harm from being done in the first place.\nPrevent access to files In our example, we use nginx to host a WordPress instance. With the right instructions, we can tell nginx to block all files with a specific file extension. In the case of WordPress installations, we don\u0026rsquo;t want to allow any access to backup files, SQL dumps, nor text files. So first thing is to configure a rule which blocks this:\nlocation ~* .(?:bak|MD|sql|tar.gz|tgz|txt) { return 404; }\n_Explanation: _search within all requests (~*) if something ends on .bak or .MD, etc. Any dots should be escaped, as they otherwise become part of a single character match within the regular expression. Upon a match, we return a 404 (file not found) error for that particular request. We use a 404 code, to indicate there is no such file. This way we don\u0026rsquo;t disclose there is a file, even if it actually available.\nWith the block above, we also restrict access to a common file robots.txt, often at the root of the website. As more specific matches have preference, we can tell nginx to allow this file.\nlocation = /robots.txt { return 200 \u0026#34;User-agent: *\\nAllow: /\\n\u0026#34;; } In this example, we return directly the file contents, instead of getting it from disk. You can also pass the request, and try to fetch it from disk, or the related WordPress installation behind it.\nWordPress Security needs multi-layering Information disclosure is just one of the many facets of information security. So if you want to securely host your website or blog with WordPress, consider taking the right steps on different layers. This way your WordPress security level can be boosted.\nThe example with blocking access to text files shows that defense in depth is needed. If you reinstall or update a plugin, the earlier deleted files will be back. So for each step you need to be on guard that things can reappear. Filtering out these files will help at least to prevent disclosing them later in the process.\nSecurity is an ongoing process. So doing one action is fine, but not enough. You should continuously think about the next step to further improve the defenses of your WordPress installation. If you never did some WordPress security hardening before, here are some quick tips to look into, to prevent your WordPress installation from being hacked.\nTips for webmasters and authors:\nSubscribe to notifications, or follow blog feeds with a RSS reader Perform regular updates Check the origin of the plugins Delete unneeded plugins Check Google webmaster tools to monitor for malware infections More advanced tips:\nIf you have PHP knowledge, check what plugins do, before installing them Create a cronjob to delete text files in your WordPress installations Run a web application firewall (e.g. Naxsi for nginx) Perform a vulnerability scan Got other tips to decrease the information disclosure of WordPress installations? Let it know!\n","permalink":"https://linux-audit.com/hardening-wordpress-security-reduce-information-disclosure/","tags":["information disclosure","web"],"title":"Hardening WordPress Security and Reduce Information Disclosure"},{"categories":["Authentication","Passwords","SSH","System Administration"],"contents":"In need of support from a colleague or vendor, but don\u0026rsquo;t want to give them permanent access? SSH has an option to allow temporary access! Next time you need to provide temporary access for an hour or day, use this great option.\nConfiguration We have two machines for this purpose. One is a system running Arch Linux, the client system. The other one is a server, running Ubuntu Linux. For temporary support, we have created a functional account support on the Ubuntu server. In the examples along the road, user michael is the one providing the support. So we are going to give him access to the support account. Temporarily!\nSuggestion: On each of the machines running commands, set your umask correctly (e.g. umask 077). Otherwise, files will be created with loose permissions, and result in errors later on.\nCreating a Certificate Authority The first step is to create a CA key. This key will be used to sign the public key of the user providing the support. Ideally, this key creation should be done on a secure system.\nssh-keygen -f ssh_ca\nThis should result in two files:\nssh_ca (private key) ssh_ca.pub (public key) Want to have a more secure key? Use the -b parameter and increase it to 4096 bits.\nThe signing process After creating the CA key pair, it is time to sign the user public key with the CA key.\nCreate an SSH key pair for users If you don\u0026rsquo;t have an SSH key pair for your user account, create one first.\nssh-keygen -t ed25519 -C \u0026quot;my key for purpose X\u0026quot;\nSigning the user key Now we need to copy the public key of the user, to our system which holds the CA key. This way we can sign the public key. Use SCP or e-mail to transfer it to the machine. For our demo purposes, we will perform all the actions on the same system. Don\u0026rsquo;t do this in production and keep keys properly separated.\nTime to do the signing. In this example, we use an Ed25519 public key. Replace it with id_rsa.pub if you used a RSA key.\nssh-keygen -s ssh_ca -I michael -n support -V +1d ~/.ssh/id_ed25519.pub\nSo what does this command do?\nParameter Purpose -s key Sign a particular key -I Define key identity -n Principal, the name of the user or host -v +1d Allow for one day The output will look something like:\nSigned user key /home/michael/.ssh/id_ed25519-cert.pub: id \u0026#34;michael\u0026#34; serial 0 for ubuntu valid from 2015-12-23T14:03:00 to 2015-12-24T14:04:10 So we see the user key is signed, and a new file is created (id_ed25519-cert.pub). We can query details about this key with the same ssh-keygen utility.\nssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub\nThe output will show something like this:\nSSH user certificate for temporary access\nNote: For demo purposes, we tried using a non-existing username (ubuntu). This is the principle listed above. By providing an incorrect principle, access will be denied. So ensure that you pick the right principle.\nReturning the signed key With the public key signed, share this new file (id_ed25519-cert.pub) with the user, so he or she can use it for logging in.\nTesting authentication with temporary access So now we have signed the key with our CA key and set a validity. Time to log in!\nssh -v support@192.168.1.223\nThat doesn\u0026rsquo;t work\u0026hellip;\ndebug1: Next authentication method: publickey debug1: Offering ED25519-CERT public key: .ssh/id_ed25519-cert.pub debug1: Authentications that can continue: publickey,password debug1: Next authentication method: password Our public key (signed by the CA) was offered, but not accepted as a valid authentication method. SSH continued with the password option, which we don\u0026rsquo;t have.\nTo get things working, we have to add the public key to the other end. However, we don\u0026rsquo;t want to allow the public key to have permanent access. So instead, we add the public key of the certificate authority.\nConfiguration on server The first step is to configure the account on the receiving server. In our case the support user.\numask 700 mkdir /home/support/.ssh touch /home/support/.ssh/authorized_keys Add then the CA public key to the authorized_keys file.\ncert-authority ssh-rsa AAAAB3NzaC1yc2EAAAA\u0026lt;long string\u0026gt; Ensure that you are copying the public key of the certificate authority. We want to trust only those authentication requests, which are signed by our CA.\nLogging in Now let\u0026rsquo;s try again and see if it works.\nTemporary SSH access granted and later denied\nAccess granted (and denied) This time authentication succeeds and we are greeted with a message of the day.\nWe can also see at the bottom of the screenshot that the second attempt failed. This is because we tried logging in after the end date of the signed certificate.\nThe related debugging of a successful login:\ndebug1: Authentications that can continue: publickey,password debug1: Next authentication method: publickey debug1: Trying private key: /home/michael/.ssh/id_rsa debug1: Trying private key: /home/michael/.ssh/id_dsa debug1: Trying private key: /home/michael/.ssh/id_ecdsa debug1: Offering ED25519 public key: /home/michael/.ssh/id_ed25519 debug2: we sent a publickey packet, wait for reply debug1: Authentications that can continue: publickey,password debug1: Offering ED25519-CERT public key: /home/michael/.ssh/id_ed25519 debug2: we sent a publickey packet, wait for reply debug1: Server accepts key: pkalg ssh-ed25519-cert-v01@openssh.com blen 862 debug2: input_userauth_pk_ok: fp SHA256:Ula18qianKQgqdfEkxRG8dK5EtaV5xyOiWdy4GAuodE debug1: Authentication succeeded (publickey). And the same request for the expired attempt which took place later:\ndebug1: Next authentication method: publickey debug1: Trying private key: /home/michael/.ssh/id_rsa debug1: Trying private key: /home/michael/.ssh/id_dsa debug1: Trying private key: /home/michael/.ssh/id_ecdsa debug1: Offering ED25519 public key: /home/michael/.ssh/id_ed25519 debug2: we sent a publickey packet, wait for reply debug1: Authentications that can continue: publickey,password debug1: Offering ED25519-CERT public key: /home/michael/.ssh/id_ed25519 debug2: we sent a publickey packet, wait for reply debug1: Authentications that can continue: publickey,password debug2: we did not send a packet, disable method debug1: Next authentication method: password The server will show (in the last attempt) that the certificate is expired. Great, that proofs it is working like intended.\nDec 23 16:16:56 ubuntu1404 sshd[2087]: error: Certificate invalid: expired\nTroubleshooting common errors When using SSH keys, the smallest things can prevent things from working. As you are working with private and public keys, ensure that you are working with the right key. Also set file permissions tight, to prevent SSH from bailing out.\nCheck the server log (e.g. /var/log/auth.log) Check file permissions Run ssh with -v or -vv Check system time Error: sshd[1381]: error: Certificate invalid: name is not a listed principal While signing the key, ensure that the principal is correct. This is the -n parameter during the key signing process.\nUnprotected private key file Make sure that file permissions are set correctly. The easiest way to have strict permissions is by defining a umask 077, so files are created with octal permissions 600 and directories with 700.\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0644 for \u0026#39;.ssh/id_ed25519-cert.pub\u0026#39; are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. Conclusion The configuration and options of SSH are very powerful. This gem is not commonly used, but very powerful to restrict access. It might be a great option to provide temporary access during holidays, or when an external party needs access for just one day. In upcoming blog posts we will dive deeper into the other options.\nRead the OpenSSH security and hardening guide article for more tips.\nDid you learn something from this article or got feedback? Let it know!\n","permalink":"https://linux-audit.com/ssh/granting-temporary-access-to-servers-using-signed-ssh-keys/","tags":["authentication","authorized_keys","hardening","ssh","ssh-keygen","umask"],"title":"Granting temporary access to your servers (using signed SSH keys)"},{"categories":["Authentication"],"contents":"When you want to allow public key authentication, you have to first create a SSH keypair. Next step is then the distribution of the public key to the other systems. Let\u0026rsquo;s have a look at a few options, including using the ssh-copy-id utility.\nOption 1: Manually In the past, you had to log in manually to the new system and do things yourself. Especially if you created your key with a tool like PuTTYgen on Windows. Then you logged in on the other system and created a .ssh directory and the related authorized_keys file. Of course, it was common to forget setting the right permissions, resulting in the authentication failing.\nOption 2: Using ssh-copy-id It is much easier to use the SSH utility ssh-copy-id. Just run the tool and provide it with your username on the remote server, with the remote server name.\nssh-copy-id michael@my-server\nIt will use your local environment to determine the related key(s) and copy it over. In case you use an alternative identity file, you can provide that with the -i option. Same for when running on a different port, specify it together with -p. To simplify your life, set up a ssh_config file. This way the right username and port are used.\nDeploying an SSH key with the ssh-copy-id command\nOption 3: Script it yourself If you don\u0026rsquo;t have the ssh-copy-id, or using Windows, you have to create something yourself.\nThe piece of magic needed:\numask 077; test -d .ssh || mkdir .ssh ; cat FILE-WITH-PUBKEY \u0026gt;\u0026gt; .ssh/authorized_keys\nThis sets your umask, so files created will be with file permission 600, and directories with 700. Then the test function has a look if the .ssh directory exists. If not, it gets created. Last step is adding your key to the authorized_keys file.\nWindows If you are on a system running Windows, use can leverage the plink utility.\ntype public_identity_string | plink.exe -pw username@hostname \u0026quot;umask 077; test -d .ssh || mkdir .ssh ; cat FILE-WITH-PUBKEY \u0026gt;\u0026gt; .ssh/authorized_keys\u0026quot;\nEnsure the right string goes into your authorized_keys file. Otherwise, logging in won\u0026rsquo;t work.\nBonus option: Fully automated If you manage a lot of systems, all these steps don\u0026rsquo;t make sense. It is better to automate the key distribution for users. This way new users get access to the right systems, while old employees have their access revoked at the moment they leave the company.\nOne of the possibilities to distribute keys is with a configuration management solution, like Ansible, Chef, Puppet, Salt, etc.\n","permalink":"https://linux-audit.com/ssh/distributing-ssh-keys-using-ssh-copy-id-manually-automated/","tags":["authorized_keys","automation","ssh","ssh-copy-id","ssh_config"],"title":"Distributing SSH keys: using ssh-copy-id, manually, or automated"},{"categories":["Passwords"],"contents":"As system administrators, we know we sometimes have to create passwords. It might be for ourselves, or when creating them for colleagues and customers. For an easy and quick way of generating random passwords, we can use the OpenSSL utility, part of OpenSSL and LibreSSL . This toolkit is often already installed on systems running Linux.\nOpenSSL has a randomize function. If we feed the output through the base64 function, the scrambled set of characters can be made more human-friendly. This function is also used for e-mail, to store binary data safely. Besides ending up with a nice set of readable characters, the password is fairly strong as well.\nopenssl rand -base64 48\nThis way of password generation is very useful for scripts, or when you need some inspiration when handing out a temporary password. If you feel you want to use a shorter password, simply reduce the number at the end. This might be useful if people have to type it in manually.\nAnother option is extending the length and using it for a secret key (e.g. for VPN and IPSEC).\n$ openssl rand -base64 1024\nTCNfOEN1AfklWa2gJQW4j5wHcnAeTvkWW1Rh9LeumX5IXo9pvI/18BUfSXToEaWy\n1GoRpa1yePEjewwnI4nmyHqQJB6yQ5mpB07k+FUIodzwkLbSb1/lDRoOAHlxOeH2\ntNhoWnaKv+HfppxDvN92ZDvfM1QW6lanu3K84h8VpsGUYUhZYcOvdlqQ82ZyIg9V\nbz5HBBVVZwlPCE6QhFVcqfQ0WWOS8IE6ysf3XhXYlHKXxkRQLFGYthq2fi0drDPZ\n9i0EWktSDibl1rVsW/g7//VeQn+boQCig53hGuOZuJU0toUrvqoWFctnReUDh+y+\n7+T08KyVS3p+m+RrcgShpFfzbi6/aJaySfDJDd5Cgk5u8ZQ6oeCH3t1qRA8I2z1Y\nfQrxsznba1Dq+irjG+2E8LohjaUimTCBoRWgjJw7cXKzbZYhnu6CJc37yE47aLCM\n9DCoA8tIfuHGs7bxfOrYvVWQmNUptqoL+ntk/r7924US6bb0tHNYNdkXT02fhyQc\nPykclFqAp+91h53kgLIW1LdWxbc4i39ile+7YfTvOJKl3zKEaXiMjaVqJme575As\nENuFAXSkzg/tgyJSgmBGWE6ornaBvMBufPb7TwgEWD0j4KeglgpNFR9jZtmzCkdq\nzHx0XHq/1owIbtUU1i/72k8lZ0T9xUd1QQMc0ExP1XYPpqNMWKutO3qNQ/2Mvv35\nu8DGE8lhu+P7e+d8puFUh5ISnDb4FVSudH1LzjSNeYmbkLTxvLRHdntuBl+1fAZj\nqxv7STWKVzVA7pyw4St/Fef3Yvs8NZEW01sbuQ3P1+7dhz4Ut55xtJtR/z9nb7Et\nwdU0WfyU3IJuC1h2X8iVcckJYb3d83dlswpPnAXL5yUpuX3F44WT5cn12ufVRwF5\n52efWsENVypB9wDYnX1ukHfYytdcH6Pj6P9pUwIUh1ZxU9LADwsoXDMLcq/Bz6Oh\nDK7uIsXTmc3remkKdv5QDAPpLaC51E1ZW7zAZTJEkVi+7INmwOZ0c0/bBeMCHcDB\nM3VuxGTeJ3oZbb43KrHf7NVHAU3FrK/N4Fa8NGYWtmrAEXTwc4ijk8v5CcefWmJy\nJ1WXjYyOEfDNkfpAf0ZoQTtt7eBXpjPq+2zJZ2olbdmXtIY2XOXEBW57/QNRbHNW\nb5VrTnCZWZX683rJbGJkVMPRDDgxghLyXVjTUIQc1xMBW7LUdQqQTy94+toYviLs\nK9jQRgKYjTb5tGQCtQ6iGTIpeeX0t2fT6R634PoY/308rLiqREDcZ5ASs23fwkFs\n19ozVx4FZoNnI6VIi8xFkoSNU4VgizJvKWFJd2jDVH98JK28fcNbtverIzI+x/Ne\nNfoptIWVLfxeryveo13UPg==\nNow that is a nice randomized key, with enough entropy. It includes capitals, numbers and \u0026ldquo;other\u0026rdquo; characters. Sure, the set of the latter is limited, but it will definitely take huge amounts of processing power to crack this.\n","permalink":"https://linux-audit.com/create-random-passwords-with-openssl-libressl/","tags":["openssl","password","security"],"title":"Create random passwords with OpenSSL/LibreSSL"},{"categories":["Firewall"],"contents":"Most system administrators will already be familiar with iptables. It is around for quite a while and is enabled by default within the Linux kernel. We can use iptables to block one, multiple IP addresses, or even full networks. This may come in handy when you get repeating port scans or see failed login attempts in your log files. Time to get started and block some IP addresses!\nCheck existing iptables configuration The first step is to validate existing iptables rules. We will use an empty ruleset for test purposes.\niptables -L\nManually blocking a single IP address The first option to permanently block an IP address is by creating a rule in the INPUT chain. This way traffic is no longer allowed from that particular IP address.\niptables -I INPUT -s 192.168.1.100 -j DROP\nAlthough this option works great, it might not scale very well. You might even get a very long list of IP addresses to block after a while. Let\u0026rsquo;s have a look at ipset.\nUsing blocklists with iptables and ipset Another option is creating a blocklist. This way we can add multiple systems we no longer want to connect to our systems.\nInstall ipset utility Most Linux systems do not have the ipset utility installed by default. So first install that toolkit.\nCentOS yum install ipset\nYou may need to install the epel-release package first.\nDebian and Ubuntu apt install ipset\nCreating a blocklist With the newly installed ipset utility we create a new list to block IP addresses. We name it blocklist to show clearly its purpose.\nCreate blocklist with ipset utility (once)\nipset create blocklist hash:ip hashsize 4096\nNote: if you want to block based on networks, use hash:net.\nAfter the blocklist is created, we can use the set in iptables. It is related to the -match-set option.\nSet up iptables rules. Match with blocklist and drop traffic:\niptables -I INPUT -m set --match-set blocklist src -j DROP iptables -I FORWARD -m set --match-set blocklist src -j DROP These commands will add the blocklist (or set) to the INPUT and FORWARD chains. As this is a blocklist, the related policy is to drop traffic. No output will be displayed when entering the commands.\nAdding IP addresses to block Next step is adding actual IP address to the list. Add a specific IP address to your newly created blocklist:\nipset add blocklist 192.168.1.100\nShow details To confirm the blocklist contains the IP address, use the ipset list command.\nIn this screenshot, we can see the IP address is listed as a member of the set. Now traffic should be blocked.\nTest rules and activate rules on reboot When setting up a blocklist like this, always test it. You want to be sure that the blocklist is enforced in your specific configuration. Also, make sure it still works after a reboot of the system.\nTo save and restore iptables rules, use the package iptables-persistent. As the name implies, this makes the iptables rules persistent across reboots.\napt install iptables-persistent\nTo also store ipset rules, create a small systemd service file: /etc/systemd/system/save-ipset-rules.service\n# ipset save/restore script (see https://linux-audit.com/networking/iptables/blocking-ip-addresses-in-linux-with-iptables/) [Unit] Description=ipset persistent rule service Before=netfilter-persistent.service ConditionFileNotEmpty=/etc/iptables/ipset [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/ipset -exist -file /etc/iptables/ipset restore ExecStop=/sbin/ipset -file /etc/iptables/ipset save [Install] WantedBy=multi-user.target This script helps to save and restore the ipset rules. You may need to create the /etc/iptables/ipset file.\n/sbin/ipset -file /etc/iptables/ipset save\nCombining ipset and IPv6 If you want to use IPv6 addresses, create the related database with the \u0026lsquo;inet6\u0026rsquo; family.\nipset create blocklist6 hash:net hashsize 4096 family inet6\nThen create the ip6tables rule:\nip6tables -I INPUT -m set --match-set blocklist6 src -j DROP\nHappy blocking!\n","permalink":"https://linux-audit.com/networking/iptables/blocking-ip-addresses-in-linux-with-iptables/","tags":["firewall","ipset","iptables","linux","networking"],"title":"Block IP addresses in Linux with iptables"},{"categories":["System Administration"],"contents":" Sometimes programs crash, usually for a different variety of reasons. While it is good to do research and find the underlying cause, sometimes you simply want to disable any reporting.\nClean up /var/crash First thing to do is check your /var/crash directory and see if there are any \u0026ldquo;crash\u0026rdquo; files. These are just normal text files and include details about a process. If you have a process crashing regularly, you most likely want to report it, so the vendor can implement a fix.\nFor the purpose of this article we are not going to analyze them. Instead, just delete them and proceed with the next step.\nDisable Apport Ubuntu systems use Apport for this. To get rid of any reporting popups, disable the Apport tooling:\n$ sudo sed -i \u0026#39;s/^enabled=1/enabled=0/\u0026#39; /etc/default/apport $ sudo cat /etc/default/apport # set this to 0 to disable apport, or to 1 to enable it # you can temporarily override this with # sudo service apport start force_start=1 enabled=0 This sed command changes the enabled status (from 1 to 0), which prevents Apport from running next time.\nThat\u0026rsquo;s all!\n","permalink":"https://linux-audit.com/software/troubleshooting/how-to-disable-system-program-problem-detected/","tags":["linux","software","troubleshooting"],"title":"How to Disable \"System program problem detected\""},{"categories":["Vulnerabilities"],"contents":"If you worked with a computer the last decade, you know the importance of keeping your software up-to-date. Those who don\u0026rsquo;t, are stacking up vulnerabilities, waiting for them to being exploited by others. Although Linux and most software are open source and can be reviewed, security flaws in software packages remain. While it isn\u0026rsquo;t easy to close every vulnerability on your system, we can at least create a stable process around it. This guide explains what is available, from vulnerability to treatment.\nLinux Vulnerabilities What is a vulnerability? As with many technical terms, the details are in the definition. The exact definition of a vulnerability differs for every organization. However, it common to describe it as a weakness in an asset, process, or piece of software. The risk involved in having a vulnerability is that a known or unknown threats (or threat actors) might abuse the weakness. This in its turn can result in a specific bad outcome, like data loss or exposure. For example, a programming flaw has a chance to become a big data leak, with all your personal data in the hands of unauthorized individuals.\nFor this article, we discuss two main categories of vulnerabilities commonly found on Linux systems. The first category contains vulnerabilities in the operating system and software packages. The second category describes weaknesses in the configuration of software.\nCommon Linux vulnerabilities Linux has weaknesses similar to those other operating systems have. These weaknesses are inherent to how computers work. Most of them are caused during the development cycle of software. The weakness is usually somewhere in the logic involved. One missing \u0026ldquo;if\u0026rdquo; statement can be enough to make a piece of software instantly vulnerable to a common attack. The big difference is that every operating system has different ways to deal with them. This starts with the compile flags used during compiling the source code, up to the time when software is being executed.\nProgramming defects Most of the security updates provided by Linux distributions solve one or more programming defects. Ranging from buffer overflows to incorrectly handling resources, all of them have a different security risk involved. In all cases, they contain a vulnerability. The million dollar questions are who can abuse them and the impact. Some vulnerabilities can only be triggered by local users, where another one might be part of a commonly used web server.\nWeak configurations Software packages usually come with a default configuration. This configuration tells the software how it should work and what logic steps it should, or should not, apply. While most of the individual settings are functional, some of them might negatively impact security measures. The opposite is true as well, it might improve the security of the software or even system as a whole.\nExample: Django Too often default or adjusted settings result in introducing weaknesses, which we often don\u0026rsquo;t recognize at first sight. A simple option like turning on debugging functionality might give an end-user detailed knowledge about the system, or even turn off some important checks.\nThe Django framework is a great example. It has a debug option, which is turned on or off with the DEBUG setting. Even though the Django documentation warns for the impact of using the DEBUG setting, we often are tempted to temporarily disable valuable protections. Many production systems end up with settings which are only suitable for during development.\nVulnerabilities with fancy names The last few years several serious vulnerabilities found their way to the media, like:\nGHOST Heartbleed POODLE Shellshock All these fancy names simply represent a weakness in software components, varying from flaws in protocol design, up to in lacking security checks within the source code of the related programs itself. What these vulnerabilities have in common is that they had (and still have!) a serious impact on common services on the system: GHOST affects glibc, a common systems library. Heartbleed and POODLE were related to SSL/TLS, often used by implementations in network services. Last, but not least, Shellshock which is named after a flaw in Bash.\nVulnerability detection and tools While it would totally make sense to learn first about the ways to protect ourselves against vulnerabilities, we should consider the reality: vulnerabilities happen. It is better to have good detection capabilities and act appropriately, than having preventing measures without 100% guarantees and ending up with a false sense of security. Prevention is great, but prevention plus detection is much better.\nNotifications Most Linux distributions have a policy in place to describe how they deal with security related issues. Users can report security issues with the website itself, services like bug trackers, or packaged software components. Especially for this last group, notifications are often sent to a related security mailing list.\nArch Linux Security Advisories Debian Security Advisories Gentoo Announcements (including security) Ubuntu Security Notices Package audit tools Receiving notifications is great, but automation is definitely the next level. Most Linux distributions have a package manager available, to control software packages. This may be simple actions of installing a package or removing. These utilities include dpkg, rpm, named after the package format. While this is great for installing and removing, the process of upgrading is slightly more complicated. For that, Linux distributions include more advanced tools, like apt-get, YUM, or dnf. These tools maintain a package database, with the installed packages, available packages, and other supporting querying tools.\nNewer package managers have the option to retrieve updates, including packages marked which have a security update available.\nGentoo: glsa-check openSUSE: zypper lp (grep security) Fedora / RHEL: yum list-sec security Ubuntu: apt-check The package manager can easily be turned into a package auditing tool with these commands. The result is a nice overview of available security updates for your system.\nFor those running BSD systems, there is \u0026ldquo;pkg audit\u0026rdquo; on FreeBSD and \u0026ldquo;pkg_admin audit\u0026rdquo; NetBSD.\nLinux security auditing Like auditing of software packages, we can perform a more in-depth type of audit: the system audit. While an audit sounds like a formal process performed by auditors, this technical checkup can be as simple as running the right set of tools. In the past Tiger was a great tool, now it has replaced with tools like Lynis or using the Security Content Automation Protocol (SCAP).\nThe benefit of auditing tools is that they look at both your hardening measures (defenses) and flaws in your configuration (vulnerabilities). Too often people focus on only vulnerability scanning. This is a shame, as a technical audit can actually reveal underlying problems and provide better insights. A hidden issue might be a misconfigured set of DNS or NTP servers.\nLinux kernel security overview after Lynis audit.\nVulnerability scanners Another great addition for detecting vulnerabilities is (of course) using a vulnerability scanner. These tools often have predefined rules to determine weaknesses in software or their configuration. While not always having the in-depth level of host-based audits, they still provide a quick way to scan the network or a single system. If combined properly with auditing, you have a great way of detecting issues quickly.\nBesides many commercial scanners, OpenVAS is probably the most-known tool. It is a fork of version 2 of the previously open-source Nessus scanner. There are more open source vulnerability scanners available, each with their own specialization.\nVulnerability prevention and reduction Now we discussed some of the detection methods, it is time to get some preventative and corrective measures in place.\nMinimal installation The easiest way to prevent weaknesses from happening is by not installing what you don\u0026rsquo;t need. In other words, apply a minimal installation. This way you only run the minimum required processes to provide a service to your users. This actually also applies to the users on the system. Remove those users that no longer belong on the system.\nCode auditing If you perform software development yourself, a great way to prevent security flaws is using code auditors. This could be a linting utility, which catches missing variable declarations, or otherwise unexpected logic. There are also commercial and open source utilities to guide with writing more secure software. Since there are so many languages, we might cover that in a later blog post. For now, just search for your programming language and code audit (or lint), and see what common utilities exist.\nTraffic filter with a firewall Many serious vulnerabilities have a high impact because they are available via the network, which increases the risk of being abused. To counter this, firewalls are another level of defense. Configure a local firewall and only allow those services required for the needed functions of the system.\nWhile network filtering with a firewall is great, it doesn\u0026rsquo;t help with those services you opened. For example, when running a web server, access to port 80 (HTTP) and 443 (HTTPS) should be generally available. Still many services have also firewalling on that level. In this case a web application firewall (WAF), which can filter out traffic.\nMore ways to reduce risks Every step that you take should be the outcome of a well-balanced decision between benefits and cost. In the end, reducing risks will be relying on your risk appetite and the resources you have available. Part of these resources is the available knowledge about a particular subject, including Linux security measures. To better understand the available measures, have a look at the article covering security for Linux systems.\nVulnerability treatment The last level of dealing with vulnerabilities is actually treating them. Usually, this is a corrective step, as it acknowledges there is an issue and we want to solve it or limit its impact.\nSoftware upgrades The most obvious corrective measure is keeping software packages up-to-date. This way when a security leak has been found, and the vendor released an update, it can be fixed quickly. Having a good patch management process in place is key.\nUpgrading on a regular basis keeps your system protected\nInstall security updates only? If you rather don\u0026rsquo;t want to install all updates, you can opt for doing only the security updates. Depending on your Linux distribution, determine if you can enable this. Sometimes we have to script a little bit to achieve the same, but it is doable. For example, with Debian and Ubuntu, we can filter out the security-related repositories, and only upgrade packages which are referred to in our custom file.\ngrep security /etc/apt/sources.list \u0026gt; /tmp/software-security.list grep security /etc/apt/sources.list.d/* \u0026gt;\u0026gt; /tmp/software-security.list apt-get upgrade -oDir::Etc::Sourcelist=/tmp/software-security.list -s Benefits The big benefit of only applying security updates is that you solve vulnerabilities, yet don\u0026rsquo;t update or upgrade the whole system. This way it is much quicker and easier to detect if something does not work after the change. It also allows you to use a double schedule, like installing security patches daily, the remaining patches one a week or month.\nTest before you patch If you or your colleagues are scared of applying patches, then create your own test system. Do a maximum installation of your favorite Linux distribution and install all types of software, you would otherwise find also on other systems within your network. Have this system perform security updates very regularly and report on any issues.\nDetecting issues with patches can be as simple as adding the system to your monitoring system. Have it test all the processes you expect it to be running. If suddenly something stops, you can quickly determine if this is the result of an applied patch.\nAutomatic security upgrades Some Linux distributions allow also for automatic patching, so you don\u0026rsquo;t have to write your own script if automation important. For Debian and Ubuntu system there is the unattended-upgrades utility. Configure it to do security patches only and let it run daily.\nConclusion Vulnerability management is a diverse set of activities to help discover vulnerabilities, categorizing them and apply the right measures. By using the right tools, the process of detection, prevention, and correction, becomes much easier. Another thing we should not forget is that security is a process. You can\u0026rsquo;t solve security just with one single product, nor is it a one-time event. The right level of monitoring and detection capabilities are needed.\n","permalink":"https://linux-audit.com/vulnerabilities/linux-vulnerabilities-explained-from-detection-to-treatment/","tags":["auditing","debian","firewall","gentoo","linux","openvas","security updates","shellshock","software vulnerabilities","ubuntu","unattended-upgrades","vulnerabilities","vulnerability management","vulnerability scan"],"title":"Linux vulnerabilities: from detection to treatment"},{"categories":["Network"],"contents":"The network configuration is a common place to start during system configuration, security audits, and troubleshooting. It can reveal useful information like MAC and IP addresses. This guide helps you to gather this information on Linux, including listing all available network interfaces and its details.\nShow network interfaces Linux Every Linux distribution is using its own way of configuring the network configuration details. Therefore, it is good to know which tools can be used to query these details in a generic way. So these commands should be working on the popular distributions like Arch Linux, CentOS, Debian, Gentoo, RHEL, and Ubuntu.\nThe old way: ifconfig Previously the most obvious command to obtain the available network interfaces was using the ifconfig command. As some systems no longer have that command installed by default, we will also look at using alternative ip. If you still have ifconfig available, run it with the -a parameter.\nifconfig -a | grep Link\nDepending on what particular information you need, you can use grep to get you the right lines. The ifconfig command on Linux actually has the most option available, so have a look at the man page for all details.\nModern version: using the ip command Newer Linux distributions now ship only the ip command. It is advised to start using this command instead of ifconfig, as its output works better with newer machines. Especially when using containerized applications, dynamic routing, and network aliases.\nThe easiest way to see what network interfaces are available is by showing the available links.\nip link show\nLinux network interfaces with ip link show command\nAnother option to show available network interfaces is by using netstat.\nnetstat -i | column -t\nNote: the column command is optional, but provides a friendlier output for the eye.\nShow the default gateway The default gateway is the system that receives traffic for networks outside your own. On Linux systems, this gateway is typically received via DHCP or manually configured in a text configuration file.\nUsing the ip command ip route | column -t\nThe output may look like this:\ndefault via 123.12.0.1 dev eth0 onlink 10.17.0.0/16 dev eth0 proto kernel scope link src 10.17.0.3 123.12.0.0/18 dev eth0 proto kernel scope link src 123.123.0.3 With netstat The default gateway can be listed with the netstat command.\nnetstat -r\nThe output will be something like this:\nKernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface default 123.12.0.1 0.0.0.0 UG 0 0 0 eth0 10.17.0.0 \\* 255.255.0.0 U 0 0 0 eth0 123.12.0.0 \\* 255.255.192.0 U 0 0 0 eth0 The second column shows the gateway. When it lists an asterisk (*), it means it uses the default gateway.\nAIX and Solaris These two old style platforms have of course ifconfig still available. By using the -a parameter, all interfaces will be displayed.\nifconfig -a | grep \u0026quot;flags=\u0026quot;\nTo see only the interfaces which are active, add the -u (up) parameter.\nDragonBSD, FreeBSD, NetBSD On the systems running BSD, it is also the ifconfig tool that can be used.\nifconfig -l\nFrequently Asked Questions How can I see the MTU of an interface? Use the ip show link command.\nip show link \u0026lt;interface\u0026gt;\nWhat command can I use to display the default gateway on Linux? Use the ip route command to show routing information, including the default gateway and the network interface it uses.\nHow can I test if my network configuration is correct? Test if you can reach or access both devices on your network as outside of it. This way you know that your IP address and gateway is correctly set up. If you can only access remote systems by IP address, then check your name server configuration. This is typically stored in the /etc/resolv.conf file. Another useful tool to test your system, including your network configuration, is by using auditing tool Lynis. It will test for connectivity of the name servers and retrieves the most important parts of the network settings.\n","permalink":"https://linux-audit.com/list-network-interfaces-on-linux-systems-and-others/","tags":["ifconfig","ip","linux","netstat","network"],"title":"List network interfaces on Linux"},{"categories":["Compliance"],"contents":"The standard itself is very detailed. Still, it sometimes unclear on what specifically to implement and when. This guide will help with translating the PCI standard to technical security controls on Linux systems.\nThis document has the goal to help you further secure your network and pass the PCI DSS audit. It is important to note that this guide is a set of generic tips. Your IT environment might require additional security measures. Always consult your auditor when it comes to interpreting the PCI DSS requirements and how it relates to your systems and network.\nThis article is based on the current version of PCI DSS , which is now version 3.2.1 (May 2018). Discovered something outdated or incorrect? Let it know!\nFocus area: PCI DSS auditing for Debian, Ubuntu, CentOS, RHEL, and other Linux distributions Audience: system administrators, IT auditors and security professionals PCI DSS requirements for Linux systems For every compliance standard, the web is full of information about it. Still, there is a lot of confusion about how to interpret things, especially when it comes down to the details. PCI DSS is no exception here. Whenever you are a system administrator, the IT manager, or a Qualified Security Assessor (QSA), we all have different ideas regarding the details.\nHaving different ideas regarding the implementation might be caused due to the previous experiences we had. Or the opposite, lacking specific knowledge of a particular subject. Whatever the reason, it is valuable to have more detailed resources available. Unfortunately, most websites provide only the basics. Or they are simply created for marketing purposes. In particular for Linux systems, and how several sections apply to the PCI DSS standard, there isn\u0026rsquo;t much quality content available.\nWhy we wrote this guide To help people with their compliance journey, we invested a lot of time to write this guide. It is now getting filled with tips, examples, and implementation ideas. We don\u0026rsquo;t focus just on the system administrator, but also on the IT auditor. This way both parties benefit from the knowledge and can help each other to get companies becoming PCI DSS certified using Linux systems.\nThis guide itself has been written to support one of the best-known and open source auditing utility for Linux/Unix systems, named Lynis. It is the tool written by us, to help with performing security audits. While we will advise many tools in this guide, we will sometimes point out when Lynis in particular can help. Not because we want to promote it more than others, but because we know more about it.\nBefore we go into the technical details, let\u0026rsquo;s get a better picture first about PCI DSS. It is good to know why it is there, and where (and how) it is related to Linux systems.\nPayment industry Every system should have a business purpose, or at least supporting the business. This includes production systems, spare systems, or even test systems. All these systems play an important role in your day-to-day business. If you are depending on being PCI DSS compliant, you better keep your IT environment, including these machines, in a healthy condition. Your business partners, customers, and colleagues depend on it. Payments are about trust for all parties involved. By applying the right security measures, we can ensure a high level of trust, or detect and analyze when one of these measures failed.\nThe auditor: Friend or Foe? The role of the auditor has a special smell. Some technical people really dislike auditors, as they feel their work is being judged or criticized. Others like the auditor, as they openly discuss subjects with management, which was previously ignored.\nIf you are a system administrator First lesson: don\u0026rsquo;t take things personally. The auditor is there to determine how well policy and processes are being executed, combined with the effectiveness of the related technical controls. It is not about you, or how well you do your work. It is better to leverage the knowledge of the particular auditor and see how he can help you achieve more things.\nIf you are an auditor Respect the environment in which technical people have to work. Politics, management, and limited training budgets have an impact on technical people. They want to achieve their best work, with the available resources. Especially regarding Linux, things can be done in many ways. Instead of demanding some information to be available, consider the impact it may have on the day-to-day job systems administrators have. Help them to get the right resources and involve their manager. For example, the purchase of an automation or auditing tool can be of great help.\nSecurity automation Systems are like us. They need to be properly maintained to function properly. The PCI DSS standard defines many security safeguards, including specific configuration settings. You should carefully look how every security measure fits in your own security policy, but also within the technical capabilities and knowledge you have.\nBesides the tips we share in this article, we also suggest implementing the right tooling, to support you. Making manual changes is fine, doing it automated is (usually) better. Let\u0026rsquo;s cover first some guidelines regarding configuration management. This might be of great help in becoming and staying compliant with PCI DSS.\nConfiguration management During the life cycle of systems, we tend to change them step by step. This might introduce the concept of \u0026ldquo;configuration drift\u0026rdquo;. This is a fancy term for the situation in which a system is changed step by step, to a point where it is unclear what status is actually right. This becomes especially an issue when we have multiple systems, which we expect to be configured in a similar way, yet they all differ slightly: configuration drift.\nAlignment with security policies To prevent configuration drift, we need a way to keep them in a controlled state. On a technical level, we need the right tools to check, adjust and monitor changes. Having the right tools won\u0026rsquo;t cut it though. Equally important is having the right policy available, to define the \u0026lsquo;what\u0026rsquo; and the \u0026lsquo;why\u0026rsquo;.\nTo get the policy and the technical tools aligned, a baseline is of great help. This baseline, or minimum rule set, defines what is expected from the individual configuration pieces on a system. Next step is to use tools to check system configurations and report any differences found.\nPolicies Baselines Tools With the PCI DSS standard mandating a lot of individual controls to be in place, automation is key. Instead of changing each system by hand, it makes sense to invest in automation tools and enforce the policies. Such configuration management tools for Linux include Ansible, CFEngine, Chef, Puppet, and Salt.\nAutomated auditing When the right automation tools are in place for configuration management, it is time to do the same for auditing. Discrepancies between a policy (or baseline) and technical configurations should be detected as soon as possible. This usually makes it easier to correct the difference and bring it in line with the preferred state. Where configuration management tools do the correction, we need a more in-depth and specialized piece of software to do the detection. Here come auditing tools like Lynis into play. Or you could use your own internal tooling and audit scripts to perform these checks.\nRequirement 1 Networking Systems are linked together via internal or external networks, like the internet. PCI DSS recognizes the fact that network connectivity should be properly protected, especially from untrusted network segments. While most of PCI DSS requirement 1 applies to network components, some of it may also apply to Linux systems.\nFirewall configurations When a firewall is present on Linux systems itself, it is usually iptables. Newer installations might have nftables instead.\nlsmod | grep table\nAny matches might indicate if iptables (or nftables) is active as a kernel module. If so, the configuration should be checked. In the case of iptables there can be a running configuration, and one stored on disk. Preferably these have the same configuration, with the exception for dynamically learned tables. A good example might be a dynamic blacklist, which gets filled with an external tool (like fail2ban).\nInsecure services PCI section 1.1.6b states to identify insecure services. Where possibly these services and the related protocols should be disabled. It was common in the past seeing protocols sharing authentication credentials and other sensitive data, without any form of encryption. This does not make the protocol itself insecure, yet makes data susceptible for capturing by unauthorized parties. This results in an insecure service, which needs careful consideration when being used or implemented.\nSome services include limitation of its access, like using access control lists of IP filtering methods. It might make the service still being acceptable in some environments. Where possible, insecure services should be replaced with a more secure alternative. For example, all Linux distributions use now SSH by default for remote administration. A protocol like telnet should therefore no longer be used. The \u0026ldquo;r\u0026rdquo; services like rexec, rlogin, rcp, and rsh, are insecure as well.\nExamples of commonly seen unencrypted protocols, plus their secure alternatives:\nFTP (FTPS, SFTP, SCP) HTTP (HTTPS) IMAP (IMAPS) POP3 (POP3S) SNMP v1/v2 (SNMP v3) Telnet (SSH, Mosh) To determine what protocols are enabled on a system, the netstat or ss utility can be used.\nnetstat -nlp\nSystems which have no netstat utility available can use ss instead. For example to display listening TCP connections:\nss -lnt\nTo see the UDP connections, use the -u flag.\nss -lnu\nAfter gathering the list of listening services, careful consideration should be given to what services might be unneeded for the system. Disabling services is the quickest and best way to reduce the attack surface of the system.\nSome services might be listening on all interfaces (0.0.0.0 for IPv4 or :: for IPv6), while just needed locally. In that case, the related listen statement should be adjusted to 127.0.0.1 or ::1, depending if you are using IPv6.\nRequirement 2 Vendor-supplied defaults and security parameters One of the most common attack vectors is trying to use default supplied default passwords and settings. From the typical \u0026ldquo;admin:admin\u0026rdquo; combination, up to a default SNMP community string. Whenever there is a password or secret value involved, it should not be the one provided by the supplier or maintainer of the software. Instead, you should change these values to something in line with your own security policies.\nInsecure protocols The last couple of years several protocols have been proofed to be too weak for proper protection of sensitive data. This resulted for example in the POODLE attack, which made effectively SSLv3 a protocol which should be banned. Because of its usage in web services, PCI section 2.2.3 specifically states that SSLv3 and early TLS versions should no longer be used.\nApache Disable SSLv2 and SSLv3 on Apache installations by adding the SSLProtocol option, specifying which protocols NOT to use.\nSSLProtocol all -SSLv2 -SSLv3\nThe related configuration file depends on the Linux distribution.\nCentOS: /etc/httpd/conf/httpd.conf Debian: /etc/apache2/httpd.conf dovecot ssl_protocols = !SSLv2 !SSLv3\nnginx Only enable the newest TLS versions by using the ssl_protocols directive. Add it to the configuration file (e.g. /etc/nginx/nginx.conf), and apply it to the http context. See more details in the nginx SSL module .\nssl_protocols TLSv1.2 TLSv1.3;\nPostfix For the configuration of Postfix, the protocols should specifically be blocked by using an exclamation sign.\nsmtpd_tls_mandatory_protocols=!SSLv2,!SSLv3\nsmtp_tls_mandatory_protocols=!SSLv2,!SSLv3\nsmtpd_tls_protocols=!SSLv2,!SSLv3\nsmtp_tls_protocols=!SSLv2,!SSLv3\nWhen using Postfix within your environment, have a look at the Postfix hardening guide.\nOther common programs which might need attention are:\nMail daemons Download utilities like cURL and wget If you are the auditor:\nThe first step is collecting the list of installed packages and running processes. Then search for known packages and processes which have SSL/TLS used. If you really want to cover every part of the system, consider analyzing the available binaries on the system. Those binaries using SSL/TLS libraries might need specific configuration to disable older protocols. Please note that not all utilities support configuration, or simply use the settings during compilation time.\nfor I in $(find /usr/sbin -type f -print); do ldd ${I} | egrep -q \u0026quot;(ssl|tls)\u0026quot;; if [ $? -eq 0 ]; then echo ${I}; fi; done\nExample output:\nprograms-using-ssl-tls-libraries.webp\nRequirement 5 Vulnerability management Systems should be protected against malicious software components, known as malware. While getting infected with adware on a Linux system has a very small chance, it is possible that a system will end up with a backdoor or rootkit. While prevention is always a preferred option, we have to ensure that adequate detection mechanisms are in place.\nMalware scanning for Linux PCI DSS section 5.1 describes the need for an anti-virus solution. This subject is definitely controversial for Linux administrators, as viruses on Linux-based systems are rare. Still, the platform is not fully resistant to different forms of malware and the related threats. So depending on the particular goal of a system, one or multiple tools can be a good fit.\nOpen source tools Generic: ClamAV E-mail: ClamAV PHP: LMD Rootkit detection: chkrootkit or rkhunter Note: Many commercial malware scanners are available nowadays. Each scanner has their strength and weaknesses in detecting malware threats on Linux.\nClamAV One of the most commonly used malware scanners on Linux is ClamAV. Like any other anti-virus/malware solution, it should be kept up-to-date. This can be achieved by running the freshclam utility. Determining the configuration settings of ClamAV and its individual components, use the clamconf command. It also shows statistics and helps to determine if the malware definitions are up-to-date.\nOverview of most common ClamAV utilities:\nclamscan / clamdscan - client for scanning files and directories clamd - daemon process for on-demand scanning clamconf - show configuration of ClamAV components freshclam - update ClamAV malware definitions Linux Malware Detect (LMD) Another great addition to ClamAV is using LMD or Linux Malware Detect. It is released under GPLv2 and can actually leverage the scanning power of ClamAV. It can use the inotify functionality of Linux, to scan new and modified files. The focus of this malware scanner is common types seen on Linux, including PHP backdoors and rootkits. LMD is especially useful for systems running web services, like shared hosting providers.\nCommercial malware scanners Here are some examples of commercial vendors that have a scanner which might work on Linux or macOS:\nAvast Bitdefender Cylance McAfee Sophos Trend Micro Malware checklist for PCI DSS and Linux Regular scans via cronjob Proper logging (preferable to syslog) Up-to-date virus definitions More information Blog post: Using ClamAV for PCI DSS Requirement 5 ClamAV Linux Malware Detect (LMD) Requirement 6 Develop and maintain secure systems and applications\nThe requirements in this section are mostly non-technical, like determining which procedures are in use, and confirmation of their effectiveness.\nRequirement 7 Restrict access to cardholder data by business need to know\nThese are mostly non-technical requirements.\nRequirement 8 Authentication Systems need to be maintained, which requires legitimate users to have access to them. To ensure this is possible in a secure way, Linux systems have different ways to store account details and authenticate these users. The most obvious ways to access a system is local and remote (e.g. with SSH). With the help of proper account management and authentication controls, access is granted to those who need it, while blocking others without this need.\nWithin the PCI DSS requirements, there are several controls which highlight the need for proper access, protection, and logging changes. This includes system configuration files (e.g. to PAM or SSH), but also to the logging itself. In other words, we should take appropriate measures to safeguard these configurations itself as well.\nInactive accounts Unused or inactive accounts on the system might be an unneeded security risk. This kind of accounts usually exists because there was a one-time need to log in, or simply forgotten after an employee left the company. PCI describes in section 8.1.4 that accounts older than 90 days and are unused, should be removed.\nTo determine the last time a user logged in, the last command can be used. Information is stored in /var/log/wtmp or rotated files like /var/log/wtmp.1.\nNote: It is common to find the wtmp files being rotated. Requirement 10.7 specifies that information should be available (online) for at least 3 months. Ensure that enough copies are being stored on the system itself, or available on a central logging server.\nPluggable Authentication Modules (PAM) Users need a way to present themselves to a Linux system. This process is called authentication and is done using Pluggable Authentication Modules (PAM). The PCI DSS standard does not describe how PAM should be configured, but it gives several pointers regarding:\nPassword history (8.2.5a password reuse) Password strength (8.2.3a password minimum length and character types) Password lockout and release (8.1.7 lockout duration) The PAM configuration differs between Linux distributions, so it is important to make the right changes, and to the right files. Of similar importance is to perform an in-depth test, to ensure the system works as expected.\nPassword file integrity Linux uses the common password file named /etc/passwd, together with a \u0026ldquo;shadow copy\u0026rdquo; in /etc/shadow. To ensure the integrity of the system, we should consider the file integrity of these files first. PCI describes that the storage and handling of these files should be done in a secure way. What does this mean? We could start with the file permissions of these files.\nFile permissions of password files Ubuntu\n/etc/passwd (644, owner root, group root) /etc/shadow (640, owner root, group shadow) The \u0026ldquo;other\u0026rdquo; group should never have read access to the shadow file, as it contains the hashed passwords.\nSafe storage of passwords Within the requirements of PCI DSS section 8.2.1, it is stated that passwords should be properly safeguarded. Passwords should be unreadable (8.2.1b) and protected with the right cryptographic algorithms (8.2.1a).\nRelated Check: view the contents of /etc/shadow and determine if no passwords are stored cleartext. Also, analyze the second column and see if it starts with $5$ (SHA-256) or $6$ (SHA-512).\nUnique IDs Then there is the uniqueness of the IDs listed in the files. Also, the correlation between IDs in one file which should match with the IDs in the other file.\nIntegrity tests for password files A very quick and powerful method to test both /etc/passwd and /etc/shadow is by using the pwck utility. It covers a wide variety of tests:\nthe correct number of fields a unique and valid username a valid user and group identifier a valid primary group a valid home directory a valid login shell every passwd entry has a matching shadow entry every shadow entry has a matching passwd entry passwords are specified in the shadowed file shadow entries have the correct number of fields shadow entries are unique in shadow the last password changes are not in the future Command to test:\npwck -r\nThis command will show the results of each of the mentioned tests. Not all output from this command is directly a finding. For example, missing home directories are showed as well.\nPassword history To prevent users from reusing the same passwords over and over, a password history should be applied according to section 8.2.5 of the PCI standard. It should at least store the last 4 passwords, to ensure these aren\u0026rsquo;t used by the account owner.\nPrevious passwords can be stored with the help of the modules pam_unix and pam_pwhistory. Check your PAM configuration and see if it consists these modules, including the remember parameter.\nFor environments which apply LDAP, NIS or Kerberos, this configuration might not be applied to the end systems itself, but on the central authentication server.\nPassword strength Passwords are still an important piece in the authentication process. Define a minimum password length and strength.\nTwo-factor authentication Central access points to the network, like jump servers or stepping stones, should be additionally guarded. One requirement is two-factor authentication (8.3) for entry points outside the network.\nTwo-factor authentication is usually arranged via PAM. Depending on the solution, the related module needs to be installed, configured and tested. Because PAM uses \u0026ldquo;authentication stacks\u0026rdquo;, ensure that modules are properly evaluated in the right order, with the appropriate control flag, like required or sufficient.\nRelated modules: pam_google_authenticator.so pam_yubikey.so Password changes PCI section 8.2.4 states that password and passphrases should be changed every 90 days. Changing passwords on a regular basis reduces the change of successful brute forcing cracking of passwords. It also helps with determining with inactive accounts (section 8.1.4 Remove/disable inactive user\naccounts within 90 days).\nOne way to test this is by checking the shadow file.\nX=$(($(date --utc --date \u0026quot;$1\u0026quot; +%s)/86400-90)); awk -v BEFORE=\u0026quot;$X\u0026quot; -F: '{ if ($2 ~ /^\\$/ \u0026amp;\u0026amp; $3 \u0026lt; BEFORE) {print $1 }}' /etc/shadow\nHow it works:\nGather current date and convert it to days since 1 January 1970 Extract 90 days of this number Open /etc/shadow file with awk Filter entries which start with $5$ or $6$ in the password field, and are older than 90 days Logging changes of password files Changes to password files should be logged. The Linux audit framework is a great solution for this. Since there is a lot to audit, refer to audit trails section below.\nShell The shell is definitely one of the most common parts for Linux administrators. Like personal preferences, there are multiple shells available for Linux systems. The first step is to ensure that all shells are accounted for. Determine which ones are installed and via /etc/shells which ones are allowed.\nSession Timeout in Linux shells Depending on the shells available on the system, a timeout should be configured with the appropriate value. PCI section 8.1.8 (session idle timeout) mandates that after 15 minutes, or 900 seconds, an idle session is being terminated.\nOne of the files to arrange this is /etc/profile . By using typeset together with the TMOUT variable, we can determine this idle session time, resulting in automatically logging out the user.\ntypeset -r TMOUT=900\nTips for Requirement 8 on Linux Validation can usually quickly done by reviewing the PAM, SSH, and authentication system. Due to the complexity around PAM, ensure that the configuration is correct and tested.\nAutomation tip: Lynis can take care of most validation steps in requirement 8 of PCI DSS.\nPaths and files:\n/etc/pam.d /etc/ssh Requirement 10 Audit Trails PCI section 10.7 covers the need for an audit trail. On Linux we have two common options:\nNormal logging Audit events Accounting details Normal logging The most common type of normal logging on Linux is by using syslog. It is still a popular way to store information, varying from boot information, up to kernel events and software related information.\nFor Linux logging, it is important to check the \u0026ldquo;health\u0026rdquo; of the logging configuration itself. This determines what happens with events, like which events to capture and what to ignore. You don\u0026rsquo;t want to log too much. This will fill up disks and makes troubleshooting and investigation only harder. On the other hand, you don\u0026rsquo;t want to miss out on important events either.\nThe knowledge of the system administrator comes here in handy. He or she usually knows best what files are containing sensitive information or need additional protection.\nImportant areas:\nsyslog rotation systemd Log rotation Depending on how many transactions and the level of details are stored, the amount of disk space occupied by log files can be huge. Proper log rotation should be in place, without destroying previously stored data (e.g. removal or being overwritten).\nSystemd logging Newer Linux systems will be using systemd as their service manager. These systems will also be using journald, a journal logging utility. It is not yet a full replacement, as some information will be stored in both, where other information is only available in one of the two. When setting up (and auditing) a system with systemd, ensure to check the configuration of both syslog and journald.\nLinux Audit System Besides logging, the system can collect audit events. Systems running Linux usually have support for the audit framework enabled. As this a very extensive topic, we suggest following up on these individual articles:\nLinux Audit Framework 101 - Basic Rules for Configuration Tuning auditd: High Performance Linux Auditing The Linux audit framework can be used to monitor many parts of the PCI DSS requirements, like changes to files, or access to confidential data. Since it is easy to go overboard with all the things you can watch, it is highly suggested to also perform optimizing of the audit rules. As always, test them carefully, to ensure all events are properly recorded.\nAccounting Another category of data to store is accounting details. This might be used for billing, troubleshooting or for further processing later. Accounting details are usually stored for actions performed by users, like running a particular process, or the connect time to a system. You should ensure that accounting details are equally treated as logging and audit information. It can provide a valuable resource during investigations like troubleshooting or incident response on Linux.\nOutput of ac command showing connect times\nlinux-accounting-ac-command.png\nNotes If you liked this guide, please share it with your peers. This way the requirements of the PCI DSS standard can be shared across more Linux system administrator and IT auditors.\nThe PCI DSS standard, logo and some of the linked resources are copyrighted by the PCI Security Standards Council, LLC. This guide is work based on the related standard and a guideline. Before implementing security controls on systems within your PCI scope, always consult your own auditors first, to determine if the related controls are in line with the requirements.\n","permalink":"https://linux-audit.com/linux-systems-guide-to-achieve-pci-dss-compliance-and-certification/","tags":["accounting","audit","auditing","compliance","linux","pci dss","pwck","ss","ssh"],"title":"Linux guide to achieve PCI DSS compliance and certification"},{"categories":["Auditing","Automation","Software"],"contents":"Recently I saw some tweets showing up from an old friend: Tiger . Surprised to see it being promoted, as I know the tool for years, but never seen any new releases in the last years. Both are actually a shame. An outdated tool is usually of lower value. Promoting old tools might actually disappoint others and harm the initial trust in the software.\nHistory of Tiger In its day, the tool was quite good. Seeing the tool is still being reviewed on blogs, this might be a good opportunity to check out the tool (again). At the same time, it might be a good to learn about some modern alternatives to Tiger, which are more up-to-date with current technologies. Let\u0026rsquo;s start with the history first and then dive into what we can learn from the project. Tiger is still presented as the \u0026ldquo;The Unix security audit and intrusion detection tool\u0026rdquo;.\nDue to the lack of updates, the value of this statement has been going down rapidly over the years. This was a different story when security tool first was released. At the Texas University the tool was created and released for duty. It was the same time several other tools were available, like SATAN and COPS. Those working early on with Linux, will most likely remember the names.\nDevelopment Unfortunately the project developments stalled several times. The project is open about it at its website, and shows even a merger of several projects, to keep going on. Despite that, the project is now stalled again, showing no progress for years. It is very common to see this happening with open source project, usually because of decreasing interest by the developers to continue development. Combining an open source project along your own work, family and personal time, is not easy. As an open source author myself I experienced several times that the pressure of continuously maintaining a hobby project can be overwhelming. Other reasons for why open source project stall, include a declining need for such a tool. In the case of Tiger this might not have been the case, if it was continuously being developed and kept up with its latest development. It might have been as simple as the lack of properly promotion the project, resulting in the \u0026ldquo;good vibes\u0026rdquo; to continue and attract new contributors.\nTiger Alternatives Nowadays Tiger has several good alternatives, ranging from both open and closed source solutions. If you purely look at open source alternatives for Tiger, our own tool Lynis comes to mind. Another option is using OpenSCAP.\nOpenSCAP With the goal to automate security settings and apply hardening, the OpenSCAP project is getting some traction. SCAP is a protocol defined by NIST, storing security related information. The goal using this information at a later stage (e.g. auditing or hardening).\nOpenSCAP itself is licensed under the LGPL license. At the same time we see people struggle with its implementation, especially because releases are tailored to individual releases of the operating system. So if you are running the newest version of your OS, you might sometimes have to wait 6 months for support. Additionally, the main focus is Red Hat systems, including CentOS and Fedora. This is not surprising due to them being the main driver behind the project. This is also the reason why you find the software back in their Satellite product.\nFortunately, other operating systems are slowly adopting SCAP content as well, like a recent addition of OpenSCAP to Debian. The hardening profiles used with OpenSCAP are predefined. The risk is that they might not be in line with your needs, or even crippling the main purpose of the system. So beware that this is not a fire-and-forget solution. However if you have thousands of similar versions of RHEL and want them all to be the same, then OpenSCAP is definitely a great choice for you.\nLynis Lynis is available since 2007, open source and released under GPLv3. Its focus is on performing security audits, similar to Tiger. It does not apply hardening, as it recognizes that every system can be different. It does help users to detect possible weaknesses and room for improvement, yet giving the user the control to decide what changes make sense.\nThe project can be found on GitHub , ensuring people can easily submit issues and contribute to the project. This helps the project with continuous development and supporting newer technologies like Docker containers.\nTo really understand the power of Lynis, you have to look inside the source code itself. The colored output might actually look like it was written in a higher programming language like Python. The truth is that is shell script, based on the bourne shell (not to be confused with BASH!). So it runs really on all Unix-based systems, including appliances, storage devices and your Raspberry Pi.\nAnother powerful item from the tooling is that no compilation or installation is needed. This makes it great for running it on a system, without the need for installation or changing the system itself. IT auditors and security professionals really love this during their security assessments.\nMaybe the strongest reason to use Lynis is that it will run on even the newest versions of your operating system. This is because of the \u0026ldquo;opportunistic\u0026rdquo; scanning behavior of the tool. It simply tries to detect and use as much system utilities as possible, without requiring them as a dependency. This way it can always find improvements, even though is has no predefined policy.\nOpenSCAP and Lynis are both great alternatives to the now outdated Tiger tool. So if you are in need to perform compliance testing, system hardening or simply want a security checkup, give them both a try!\n","permalink":"https://linux-audit.com/tiger-is-history-long-live-modern-alternatives/","tags":["auditing","hardening","lynis","openscap","scap","system hardening"],"title":"Tiger is History, Long Live Modern Alternatives!"},{"categories":["Auditing","Automation","System Administration"],"contents":"Lately I saw a great feature request for Lynis, to detect differences between two runs of Lynis. Wouldn\u0026rsquo;t it be great to run Lynis daily and then see if anything changes and act upon those differences? While our auditing tool doesn\u0026rsquo;t have such an option itself, it is very easy to implement something and fine-tune it to your needs.\nReport Lynis has two important files to which is logs data:\n/var/log/lynis.log /var/log/lynis-report.dat The first file /var/log/lynis.log has all technical details of the audit. The report file /var/log/lynis-report.dat contains all important scan results, like warnings, suggestions, and generic system information. It is this same report file which we can use to compare two different audits!\nScript To help you out finding differences between two scheduled Lynis runs, simply leverage the report file.\nExample script\n#!/bin/sh PERFORM_DIFF=0 # Step 1: Archive file if [ -f /var/log/lynis-report.dat ]; then cp /var/log/lynis-report.dat /var/log/lynis-report-previous.dat PERFORM_DIFF=1 fi # Step 2: Here you run Lynis (e.g. as a cron job) cd /path/to/lynis ./lynis audit system --cronjob # Step 3: Perform the difference (unless it is the first time) if [ ${PERFORM_DIFF} -eq 1 ]; then DIFFERENCES=`diff --ignore-matching-lines report_datetime /var/log/lynis-report.dat /var/log/lynis-report-previous.dat` if [ $? -gt 0 ]; then echo \u0026#34;Found differences:\u0026#34; echo \u0026#34;===========================================================================\u0026#34; diff -y /var/log/lynis-report-previous.dat /var/log/lynis-report.dat | grep -v \u0026#34;report_datetime\u0026#34; echo \u0026#34;===========================================================================\u0026#34; fi fi #EOF Here is how it works in steps:\nSteps\nArchive the existing lynis-report.dat file Run Lynis (again) Compare results Just three simple steps.\nImplementing the script If you already run Lynis as a scheduled cron job, copy that file for testing and add the top and bottom section (step 1 and 3) from the example.\nNext is testing if things work like expected. So first run your script and check if execution of Lynis was successful. You can do this by checking the /var/log/lynis.log and /var/log/lynis-report.dat files.\nThen determine if it correctly copied the previous report contents to /var/log/lynis-report-previous.dat.\nIf it copied the file, change a few lines in your active report /var/log/lynis-report.dat (not the -previous file, as it will be overwritten!).\nRun the script again and see if the differences show up:\nThe diff tool found a minor change between two scans\nFound this tip helpful? Share it with others and help more people with automation.\n","permalink":"https://linux-audit.com/lynis/find-differences-between-two-daily-lynis-audits/","tags":["audit","lynis"],"title":"Find differences between two daily Lynis audits"},{"categories":["Network"],"contents":"Most network related services have to open up a network socket, so they can start listening for incoming network requests. It is common to find the TCP or UDP being used as the main communication protocol. In this article, we will check what ports are used by which Linux process.\nFind out what process is listening to a port Only one process can actively listen to a TCP or UDP port. We usually only discover this when another process is already running on a specific port, while we try to start another service:\n[emerg] 9400#0: bind() to 0.0.0.0:80 failed (98: Address already in use)\nor something like:\nnc: Address already in use\nTime to tackle which process is keeping these addresses or ports in use!\nUsing lsof to show open files and network ports Let\u0026rsquo;s start with a powerful utility named lsof. It is not always installed by default, but still a very common utility. Its name is derived from listing open files. In Linux, even a network socket is a file. So this is definitely the right utility to retrieve some useful information. LSOF will include network-related data like port numbers and process names.\nTo find open ports and the related processes, ask lsof to see the related details. We filter out all UDP ports and only want to see TCP ports that are listening to data.\nlsof -Pni | grep -E \u0026quot;(UDP|LISTEN)\u0026quot;\nThis output might look something like this:\nLsof displaying UDP ports and TCP ports in LISTEN state\nThis commands includes all UDP ports and for TCP only the ports which are actually in \u0026ldquo;LISTEN\u0026rdquo; state. Perfect to determine which process is listening to what port (or ports).\nIf you are interested in a particular port, lsof can filter by protocol and port number.\nlsof -i TCP:80\nUsing netstat to show ports and applications The netstat is another utility to determine what processes are running and what port they listen to. It is available on most systems by default, although it is replaced by ss.\nnetstat -nlp\nThe result might be looking something like this:\nNetstat showing all running processes and ports they listen to\nIn this output, we see the following details:\nColumn Information field 1 Proto 2 Recv-Q 3 Send-Q 4 Local Address 5 Foreign Address 6 State 7 PID/Program name Protocol The protocol (Proto) defines the data transfer protocol. This is typically tcp or udp for IPv4, tcp or udp6 for IPv6.\nReceive and Send queue The Recv-Q and Send-Q are the queues to for receiving or sending data. In the output they are usually zero unless a transfer occurs at that moment.\nLocal and foreign address The Local Address field specifies the IP address with the related port number. Can listen to a particular IP address (like localhost or 127.0.0.1) or on all interfaces (0.0.0.0 for IPv4). This is usually the interesting part to filter out (tip: use grep).\nState The State column is usually showing LISTEN for TCP.\nPID and program name Then finally the PID/Program name column shows the actual process that is listening to a particular network port.\nUsing the ss command f you don\u0026rsquo;t have the netstat utility available, it might have been replaced with a newer toolkit. In that case, you usually have the ss utility available (iproute2 package).\nTo use the ss tool to see on Linux which ports are used by a particular process:\nss -lpntu\nThis will show a similar output. It shows all the listening ports, limited to UDP/TCP only, not translated to hostnames to speed up the results.\nDetecting network ports for new processes Sometimes you might be running a new process and not aware of any network ports being opened. This might occur when it just quickly happens, or due to a port listening conflict. Also when running services in containers, it might be harder to know what ports need to be opened, to get it fully functioning.\nIn these cases, the strace utility is of great help. It can track a running or new process and alert for any events of your interest. To track the open request of a network port, we can monitor the related system call (syscall), which is \u0026lsquo;bind\u0026rsquo;.\nstrace -f -e trace=bind nc -l 80\nWith this command we will see quickly after execution the following line:\nbind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr(\u0026ldquo;0.0.0.0\u0026rdquo;)}, 16) = 0\nThis means it tried to open a network socket with port 80. Unfortunately, it does not show if it opens a TCP or UDP port. If we broaden the system calls a little bit, this information becomes available:\n# strace -f -e trace=network nc -lu 80 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr(\u0026#34;0.0.0.0\u0026#34;)}, 16) = 0 Conclusion That\u0026rsquo;s it for today. Linux systems are very versatile, yet sometimes need the right tool to dig into the details you want to know. With tools like lsof, netstat, ss, and strace, we can find exactly the network information we are looking for.\nDid this article help you or have a good one-liner for other readers? Let it know, to make this article even better!\n","permalink":"https://linux-audit.com/networking/audit-which-ports-are-used-by-a-linux-process/","tags":["lsof","netstat","network","ss","strace","tips"],"title":"Which Linux process is using a particular network port?"},{"categories":["Compliance"],"contents":"If you are involved with Linux security, you might already have come across SOx compliance. Usually from a sysadmins point of view, work doesn\u0026rsquo;t get easier due to these compliance requirements. Still there are some lessons we can learn, followed by a great documentary to watch about Enron.\nThe Sarbanes-Oxley Act , SOx for short, now applies to all companies who like to have their stock exchanged at the New York stock exchange (NYSE). The goal of SOx is to reduce fraud and make people responsible if they are guilty of committing to fraud. This includes jail time and high fines, to ensure fraud won\u0026rsquo;t be unpunished.\nWorldCom Most of us might remember the name WorldCom, which was the second largest telecom provider in the US. Big companies have to take huge risks to be competitive and valuable for their shareholders. To continue showing great figures and pumping up the share prices, companies may fall into less ethical practices, like accounting fraud.\nOne of the frauds discovered at WorldCom was the use of capital expenditures (CAPEX) while they were actual normal expenses. These costs (or investments) normally are used for buying assets, or increasing the value of assets. As they are investments, they can\u0026rsquo;t be applied to normal expenses.\nTo get a full history of the company, have a look at the Wikipedia WorldCom page.\nEnron Enron was a company investing and trading in the worlds of energy. It took high gambles and made great promises to investors. That was, when everyone discovered that you simply can\u0026rsquo;t always win. The documentary \u0026ldquo;Enron: The Smartest Guys in the Room \u0026rdquo; perfectly explains how companies have to cover up a single lie, by applying the next one. Definitely worth watching to get a better understanding on why fraud and unethical behavior do not belong in a healthy company. Better yet, that the punishment will be harsh for everyone involved.\nThe documentary is available on Netflix, including several documentary websites and popular video streaming sites.\nWhat can we learn? Information security is a strategical resource for companies to protect their assets. At the same time, fraud on every level in the company can result in a lack of trust and even let companies collapse. In the case of Enron we can actually discover the impact it might have on the market, on people\u0026rsquo;s jobs and retirement funds.\nAnother things we can learn is that if something looks to be good to be true, it usually is. Most of us know this saying, yet we don\u0026rsquo;t apply it always. For example the case when buying new software solutions which promise magic, or what about falling into the trap of e-mail scams. While paranoia might not be the solution, a little bit more than average can\u0026rsquo;t hurt.\nWhile SOx compliance might complicate your work, it ensures the company does business in an honest and ethical way. Additionally, it might give you the leverage to get the right security tools in place, to protect the valuable assets of the company, including your own personal details. After all, information security reaches places we can\u0026rsquo;t always see ourselves.\nBe safe and stay honest.\n","permalink":"https://linux-audit.com/sox-compliance-and-enron-the-smartest-guys-in-the-room/","tags":["compliance"],"title":"SOx compliance and Enron: The Smartest Guys in the Room"},{"categories":["Secure","System Integrity"],"contents":"Everyone who used Windows 95 or 98 in the past is familiar with the concept of failure. One crashing application was enough to bring the system to a halt. Fortunately, Linux systems have a strong foundation, including privilege separation and memory management. When things go wrong, the impact is reduced to a minimum. This is called containment.\nLinux Memory Management Memory is like your the storage capacity of your brain. Every bit should be stored properly, or otherwise you will do strange things. Linux systems have powerful memory management, to ensure that data is properly sorted and permissions are assigned. For example an ELF binary, the most common binary format on Linux, has different sections for executable code and data. Then on top of that, each section gets different permissions in memory. For example code could be marked as read-only, to prevent it being overwritten by itself or another process.\nAs you can imagine, memory management is an important area of the GNU/Linux kernel component. A single implementation mistake is the difference between a stable system, or one that crashes for no reason.\nPrivilege Separation One of the primary reasons that Linux systems are stable is the clear separation of privileges. We already have seen it in action for Linux memory management, where different structures are separated. This goes much further on other levels of the system, including what kind of functions can be performed by executables (e.g. using Linux capabilities.\nBuild for Impact Reduction When you are building systems, we can learn a valuable lesson from the containment features of Linux. Every system should be built in such a way, that when the inevitable crash occurs, the impact to our full environment is limited. This containment of failure can be achieved by using a clear separation in functions. If one function goes down, it should only have an impact to that function. Where possible indirect damage should be limited, or avoided.\nNever Fail A Little Bit Systems will fail. Linux systems, while stable from the foundation, can fail as well. The worst outcome is a system which provides its services only half. It is not down, but not really up either. When you design your web server cluster, ensure that the load is properly shared among each node. Complete it with the right amount of monitoring, so it never gets stuck in \u0026ldquo;half\u0026rdquo; operation. This could happen when it is overloaded, yet the load balancer thinks it has enough resources left. It is better to fail completely, than just a little bit.\nConclusion Everyone wants a stable system. Stability is the sum of a lot of factors combined, like privilege separation, proper memory management, and containment. To achieve a stable operating system, and system, these factors all need be in balance and correctly implemented. In upcoming blog posts, we will have a look at the more technical aspects.\n","permalink":"https://linux-audit.com/linux-security-principle-containment-of-failure/","tags":["memory"],"title":"Linux Security Principle: Containment of Failure"},{"categories":["Software","System Administration","System Integrity"],"contents":"If you are in the business of system administration, you know the big dilemma when it comes to installing software: missing packages. Yes, a lot of packages are available in the repositories of your Linux distribution, but not the one you need. Or when it is, it is horribly outdated. So you reach out to external resources, like community maintained repositories, right?\nWith Lynis, we face this same issue. While most of the distributions have Lynis in the repository, it is often outdated. We could do packaging ourselves, and most likely will in the future. But for now, that task is taking too much time with the regular updates we provide. Packaging, testing, and checking is a delicate process, often better done by people who know that specific Linux distribution from the inside out.\nMany software components are facing the same and other people step up to provide community maintained repositories. In this article, we have a look at the benefits, but also the serious risks involved.\nThe Trust Issue One of the big problems with external resources is that you have to trust people. People you possibly don\u0026rsquo;t even know. Every single person is a new line of trust, adding up slightly more risk. That is totally fine, until you have too many \u0026ldquo;trust relations\u0026rdquo; going on. It is hard to keep up if everyone in the chain remains trustworthy. Or even worse, if someone in the chain goes bad and malicious activity occurs on purpose. In this case that might be an altered package, or a hacked software repository.\nSo in any case, you want to trust as less as possible. For those areas you want to trust, you want to have assurance that the people (or companies) involved are doing everything they can to minimize risks and maximize protection.\nWhy Not Use External Repositories Depending on your environment, packages maintained by a third party might introduce a new level of risks. For example when your environment is totally build up with RHEL systems, chances are big you need sooner or later an external component. By adding the repository, you might lose support or even face unstable software. There is also a serious risk in inadequate support to keep up with security bulletins. Voluntary repositories often don\u0026rsquo;t have the resources like the Linux distributions themselves. The only exception of using an external repository might be for official vendor supported software. An example is that of Docker. They have their own build process and release schedule, so they don\u0026rsquo;t want to rely on all Linux distributions to keep up.\nGreat, What Then? The best option is to build some software yourself, especially if you have the intention to roll it out in your whole Linux environment. This gives you the opportunity to decide what versions to use and quickly patch it when needed. By compiling and packaging the software packages, you feel also more responsible for introducing new software components. After all, a healthy barrier will be added, which will avoid you from just installing more and more external software components.\nThe Way Back So you might think \u0026ldquo;great, but I already have those external packages in my environment\u0026rdquo;. In that case not all hope is lost. Even if you use(d) repositories like those of Dag Wieers or Repoforge, things can be improved step by step:\nMake an inventory of all used repositories Craft a list of \u0026ldquo;alien\u0026rdquo; packages Determine exceptions Remove unneeded packages Build replacement packages Limit access to repositories So let\u0026rsquo;s go into more details on how to achieve these steps. The first action is getting an inventory of all used repositories on the systems. Make a shift between the native built-in repositories, and those externally hosted.\nNext step is to search for all packages and determine to which repository they belong. Were they being part of the built-in repository, or one of the external ones?\nAlien packages\nWhen you have the list of alien packages, it is time to determine which ones are really needed, and the ones which are optional. Everything unneeded should be uninstalled. The remaining packages are for the self-packaging list. Depending on your needs, this list might actually be lower than initially thought. It is common to find the same packages being installed on many systems.\nPackaging\nNext step is building the packages yourself. The first time it might be a daunting task. The positive news is that usually the external repositories often provide you the source build files. This way you can reproduce what they have done and do it yourself. Use it to build the packages and start testing deployment.\nExterminate\nThen finally when everything is done, ensure that external repositories become the past. Monitor your YUM/APT configuration files and block the addition of any new repositories. You might even want to filter them out in your proxy or firewall.\nPatch!\nLast but not least, keep your own packages up-to-date. Especially network services might be extra vulnerable for attacks from outside. Also implement a software patching plan, in case you didn\u0026rsquo;t have that yet. Security patches are released on a daily basis, so keep all packages up-to-date.\nConclusion External packages are often used to overcome the \u0026ldquo;missing packages issue\u0026rdquo;. Your favorite repositories might not be hosting them, so external ones are being used. While this might sound as an easy way, it introduces the risk of unstable software, vulnerable or even malicious harmful software.\nThe best option is limit to official repositories from your Linux distributions and well supported external vendors. They have both the capacity and knowledge to supply packages, as they name is on the line. Don\u0026rsquo;t trust external resources too much and avoid them as much as possible.\nAnd as usual, it is easy to introduce something, but getting rid of it might be close to impossible. We all those \u0026ldquo;it is just temporary\u0026rdquo; installations. Simply do not sacrifice the integrity of your software for convenience. Keep your IT environment healthy, instead of building it on all kind of loose ends and external dependencies.\nStay safe!\n","permalink":"https://linux-audit.com/missing-packages-do-not-trust-external-repositories/","tags":["software"],"title":"Missing packages: Don’t trust external repositories!"},{"categories":["Hardening"],"contents":"Create a Linux security fortress; implementing security defenses using towers, bridges, and guards.\nStill many companies have difficulties implementing basic security measures. Even after years of websites being defaced, and customer records stolen, the same mistakes are made over and over again. While this all might sound like an unsolvable situation, information security is getting attention from more people. If you are responsible for the system management of Linux systems, ignoring security is no longer an option.\nThe issue with security is that you can measure insecurity, yet not properly measure the level of security. This leads to a situation in which companies simply not knowing what to do, or when it is enough. Still by applying a few basic principles, we can fortify our systems and make our defenses more resistant against common attacks.\nRisk Management Security boils down to understanding risk. From management level, down to the system administrator, everyone is in control of some aspects of risk. We might choose to accept risks (do nothing), reduce them (implement measures), or move them to others (e.g. insurance). Finally, we can decide to skip risk, and not pursuit some action at all. These principles of risk management also apply to our Linux systems. It requires understanding of risks and threats, to allow us selecting the right measures and enhance our existing defenses.\nIn the world of IT, ignoring common threats like malware and exploiting software weaknesses is usually no longer an option. Knowing risks and threats is what makes us well informed, resulting in making better decisions and spending our precious time more wisely.\nLinux and Security Risks Like any operating system, Linux also has threats which might badly impact the confidentiality, integrity and availability of our data. The chance to find a trojan horse on the system is lower than on a Windows system, but the risk is still there. To counter threats to our precious Linux systems, we can very well compare them with a fortress. Like any good fortress, it needs to be designed, build and maintained properly. So let\u0026rsquo;s move on and let our Linux systems be equal to building a fortress.\nBuilding the Fortress To build a fortress, you will need strong towers. They act as a defensive measure and increase the strength of the overall structure. On top of that, they help with monitoring the environment. Consider the towers as your primary goals, the walls as normal ongoing business (deploying systems, monitoring, adding/removing users, etc).\nA fortress does not only exist of walls and towers. There are guards to monitor, and bridges to make something possible (e.g. cross over).\nTower 1: System Hardening The first tower is strongly related to system deployment. When installing Linux systems, go for system hardening at day 1. This can be achieved by only doing a \u0026ldquo;minimal installation\u0026rdquo;, to reduce the fingerprint of the system. It saves installation time, storage space and limits the amount of possible weaknesses.\nSystem hardening is not something you just do at installation time. There is the post-installation phase, in which you start enabling new services, like deploying your favorite monitor tool. Keep your post-installation tidy and clean.\nGuards (monitoring):\nUse automation tools like Ansible, Cfengine, Chef and Puppet Bridges (enhance):\nAutomate your (post-)installation process Minimal installations Remove unneeded components Tower 2: Software Patching The second tower focuses on software components. After installation, software packages need to be maintained. Software is like the bricks in the walls. If you don\u0026rsquo;t maintain them, they crack open and introduce additional weaknesses.\nUnfortunately still many companies fail to properly keep software up-to-date. Administrators are scared to implement patches, due to the chance of things end up broken. Good testing helps with reducing this risk, while keeping the fortress stable.\nGuards:\nSoftware version monitoring Vulnerability scanning Bridges:\nSoftware patching solution Build/test platform for (automatic) security patching Tower 3: Integrity Checking Next tower consists of performing integrity checking. Like a fortress, we should ensure that unexpected parts are quickly discovered. In this case, it could be an unknown guard among our own troops, or malfunctioning chains to open and close the central bridge. Comparing this to our Linux system, a guard could be a process or binary on disk. Do some of them look strange or are they replaced with different files? It might be the work of a digital intruder. Similar to the bridge, common processes which malfunction and crash might be showing the signs of bad system integrity.\nGuards:\nImplement file integrity monitoring with tools like AIDE Check for malware (ClamAV, OSSEC, rkhunter) Bridges:\nKeep software packages up-to-date Perform sometimes a system reboot Don\u0026rsquo;t use external components if not really needed for proper functioning Tower 4: System Auditing Like guards patrolling the fortress, and scouts doing field work, we should also check our systems on a regular basis. Consider it health checks, to ensure our measures are still working. For a fortress, it could be lifting the bridge and inspecting the chains. Or checking the food supply, for times when resources will be scarce. In the world of Linux systems, we have to check our software configurations. Check if main processes are still running as expected, and log files created and filled properly.\nGuards:\nReview log files Check software configurations Have an external auditor or colleague do an analysis Bridges:\nImplement continuous auditing and monitoring tools (scripts, Lynis) Implement system hardening Centralized syslog server Conclusion Linux systems can be fortified to reduce the most common attacks. Internal and external attackers can quickly weaken your defenses. From patch management to regular audits, integrity checking, and system hardening, they are all needed to form the pillars of a healthy construction. Your Linux system is not very different from the fortress of the medieval times.\nGood luck with building your digital fortress and keep your security defenses strong!\n","permalink":"https://linux-audit.com/security-defenses-to-fortify-your-linux-systems/","tags":["auditing","file integrity","file integrity monitoring","hardening","monitoring","syslog","system hardening"],"title":"Security Defenses to Fortify your Linux Systems"},{"categories":["Web"],"contents":"The end of Adobe\u0026rsquo;s Flash Player is near. Most of the remaining Flash on the web are advertisements or \u0026ldquo;fancy\u0026rdquo; movies, created years ago. If you don\u0026rsquo;t need Flash any longer, these steps help you to disable it in Chrome.\nStep 1: Open plugins Go to chrome://plugins\nThis will show an overview of all your plugins.\nStep 2: Disable Abode Flash Player Press Disable on the Adobe Flash Player.\nAdobe Flash Player is disabled\nThe color of the plugin changes and you are set. Seeing the active stance Google is taking on some matters, it probably won\u0026rsquo;t take long till Flash is fully gone.\nBe safe and share this tip with others to benefit from a safer internet.\n","permalink":"https://linux-audit.com/quick-tip-disable-adobe-flash-player-in-chrome/","tags":["web","web browser"],"title":"Quick Tip: Disable Adobe Flash Player in Chrome"},{"categories":["Auditing","Cheat Sheets","Performance"],"contents":"The strace utility is very powerful to learn what a new or running process is doing. Due to its diversity of monitoring options, the tool is less accessible at first. This strace cheat sheet helps with getting the best out of this tool.\nNormally cheat sheets come in a single 1 page PDF. In this case, we combined it all within a blog post. First section shows an explanation per area, the bottom of the post contains all useful commands for quick reference.\nCommonly used strace options Strace has many options, so here is a list of options that are typically can be found.\nLong option Short option Intended action --summary-only -c Report a summary on program exit and include counts for time, errors, calls for each system call --follow-forks -f Track process including forked child processes --output=FILENAME -o FILENAME Log strace output to a file --attach=PID -p PID Track a process by PID --trace-path=PATH -P PATH Track a process when interacting with specified path --syscall-times -T Display times in the output Troubleshooting with strace One of options of the strace utility is to help as a troubleshooting utility. If you want to know what a process is doing, or why it hangs, strace will definitely help. By running strace without any parameters, it will already show why a process is doing. You can trace a running process, or instruct strace to start it for you.\nAll syscall listed by amount of time\nMonitoring File activity Strace can monitor file related activity. There are two useful parts. The first is file, which shows file interactions. The other one allows tracing file descriptors. Both can be used to monitor for actions like opening files, reading/writing and closing. Usually using \u0026ldquo;trace=file\u0026rdquo; provides enough insights. If you really need more insights in the way a program deals with file descriptors, then use the second one.\nMonitor opening of files: strace -e open -p 1234 See all file activity: strace -e trace=file -p 1234 or strace -e trace=desc -p 1234 If you want to track specific paths, use 1 or more times the -P parameter, following by the path.\n# strace -P /etc/cups -p 2261 Process 2261 attached - SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=6149, si_uid=0} - lstat(\u0026#34;/etc/cups\u0026#34;, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 openat(AT_FDCWD, \u0026#34;/etc/cups\u0026#34;, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7 getdents(7, /* 11 entries */, 32768) = 336 getdents(7, /* 0 entries */, 32768) = 0 close(7) = 0 openat(AT_FDCWD, \u0026#34;/etc/cups\u0026#34;, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7 getdents(7, /* 11 entries */, 32768) = 336 getdents(7, /* 0 entries */, 32768) = 0 close(7) = 0 Common calls:\nSyscall Intended goal access Checks whether the calling process can access the pathname, dereferenced when it is a symbolic link close Close file descriptor fchmod Same as chmod, but used file by open file descriptor fd fchown Changes ownership of file, referred to by open file descriptor (fd) fstat Similar to stat(), but uses file descriptor fd lseek Reposition file offset for read/write open Opens file specified by pathname to allow reading or writing data read Read from file descriptor statfs Returns information about mounted file system See the syscalls overview for others.\nA related example screen output:\nMonitoring file access and activity with strace\nNetwork-related actions Strace definitely can be useful for revealing more details about network traffic. Very useful to determine what network related connections are used, like when building your Docker image.\nstrace -e trace=network\nCommon syscalls:\naccept(2) bind(2) getsockopt(2) listen(2) socket(2) setsockopt(2) Memory calls To get better insights on the memory usage and system calls, strace can monitor for these as well. They are nicely grouped in the memory group.\nstrace -e trace=memory\nCommon syscalls:\nmmap(2) munmap(2) Useful system call groups for tracing Track by specific system call group\nStrace syscall group Action performed -e trace=ipc Track communication between processes (IPC) -e trace=memory Track memory syscalls -e trace=network Track network syscalls -e trace=process Track process calls (like fork, exec) -e trace=signal Track process signal handling (like HUP, exit) -e trace=file Track file related syscalls Want to trace multiple syscalls instead of a full group? Combine them by specifying them directly instead of the syscall group.\nstrace -e open,close\nGot other clever stracing tips? Let it know!\n","permalink":"https://linux-audit.com/cheat-sheets/strace/","tags":["cheatsheet","debugging","howto","linux","ipc","strace","troubleshooting"],"title":"strace cheat sheet"},{"categories":["Auditing","Kernel","Software","System Administration"],"contents":"Processes are the running workforce on a Linux system. Each process has a particular goal, like forking child processes, handling incoming user requests of monitoring other processes. As a system administrator or IT auditor, you might want to know at some point what disk activity occurs in a process. In this article, we have a look at a few options to quickly reveal what is occurring in a process, including disk and file activity.\nMonitor syscalls The kernel uses system calls, or syscalls for short. These are specific functions, which perform a low-level system function. Think of activities like reserving a memory section, or in this case opening a file from disk. The first utility to provide insights in active syscalls, is the strace utility. By tracking the right system call, we can see exactly what files are opened while it happens. Great for tracking required file access, dependencies, and troubleshooting purposes.\nThe usage of strace is simple. Just run a command you normally would execute, prepended with the strace utility.\nstrace ls\nIf you run the same command on the CUPS daemon, this would be the output:\nLots of output after starting a strace on a running process\nWhile this provides interesting information, it might actually flood your screen, making it hard to work with. As we are interested in file access, we want to see only the open syscall.\nstrace -f -e open ls 2\u0026gt;\u0026amp;1\nSo let\u0026rsquo;s first check what chain does: start strace, track forked childs (-f) for the open system call (-e open). As command we track the ls utility and redirect any errors to the screen output. For other interesting system calls, see the man 2 syscalls page.\nIf you want a clean output which only shows , here is a trick to only list the files:\nstrace -f -e open ls 2\u0026gt;\u0026amp;1 | grep ^open\\( | grep \u0026quot;[[:digit:]]\\+$\u0026quot; | cut -d\\\u0026quot; -f2\nWe can also apply monitoring system calls to a running process. Provide the -p and define the process ID you want to monitor. If you also want to monitor any forked child processes like in previous example, add the -f parameter.\nstrace -f -p 4121\nMost systems have the strace utility already installed by default. If you have a minimal installation without it, use your package manager.\nTracking system calls The second option to check what system calls are used, is by monitoring the libraries used. Libraries are similar to a toolbox, filled with individual functional tools. In the case of Linux, the library is filled with functions, including indirect system functions.\nMonitoring these functions can be done with the ltrace utility. Its usage is similar strace, but with the focus on libraries.\nTo get a first impression what kind of functions are used, use the -c parameter. It lists the functions, how often it was used (calls) and the time involved with that function. Great for troubleshooting why a process is taking a while to respond.\nltrace -c ls\nAn example output of a trace on a Chrome process:\nUsing ltrace to track system calls (syscalls)\nLike strace we can attach to a process. For example tracking what the cron process does:\n# ltrace -p 1275 time(0) = 1435911661 localtime(0x60b300) = 0x7f36b92ecde0 __xstat(1, \u0026#34;crontabs\u0026#34;, 0x7fffb2700c60) = 0 __xstat(1, \u0026#34;/etc/crontab\u0026#34;, 0x7fffb2700cf0) = 0 __xstat(1, \u0026#34;/etc/cron.d\u0026#34;, 0x7fffb2700d80) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 16 __xstat(1, \u0026#34;/etc/cron.d/php5\u0026#34;, 0x7fffb2700e10) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 19 __xstat(1, \u0026#34;/etc/cron.d/anacron\u0026#34;, 0x7fffb2700e10) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 23 __xstat(1, \u0026#34;/etc/cron.d/amavisd-new\u0026#34;, 0x7fffb2700e10) = 0 gmtime(0x7fffb2702fa8) = 0x7f36b92ecde0 time(0) = 1435911661 sleep(60 It will do nothing for a while and suddenly it shows up. It looks in several common cron related files (like /etc/crontab and /etc/cron.d). The __xstat function in this case monitors the files and tries avoiding opening each of them, unless it file meta information changed (e.g. modification date). The output suddenly looks different:\n# ltrace -p 1275 time(0) = 1435912201 localtime(0x60b300) = 0x7f36b92ecde0 __xstat(1, \u0026#34;crontabs\u0026#34;, 0x7fffb2700c60) = 0 __xstat(1, \u0026#34;/etc/crontab\u0026#34;, 0x7fffb2700cf0) = 0 __xstat(1, \u0026#34;/etc/cron.d\u0026#34;, 0x7fffb2700d80) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 16 __xstat(1, \u0026#34;/etc/cron.d/php5\u0026#34;, 0x7fffb2700e10) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 19 __xstat(1, \u0026#34;/etc/cron.d/anacron\u0026#34;, 0x7fffb2700e10) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 23 __xstat(1, \u0026#34;/etc/cron.d/amavisd-new\u0026#34;, 0x7fffb2700e10) = 0 gmtime(0x7fffb2702fa8) = 0x7f36b92ecde0 time(0) = 1435912201 sleep(60) = 0 time(0) = 1435912261 localtime(0x60b300) = 0x7f36b92ecde0 __xstat(1, \u0026#34;crontabs\u0026#34;, 0x7fffb2700c60) = 0 __xstat(1, \u0026#34;/etc/crontab\u0026#34;, 0x7fffb2700cf0) = 0 __xstat(1, \u0026#34;/etc/cron.d\u0026#34;, 0x7fffb2700d80) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 16 __xstat(1, \u0026#34;/etc/cron.d/php5\u0026#34;, 0x7fffb2700e10) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 19 __xstat(1, \u0026#34;/etc/cron.d/anacron\u0026#34;, 0x7fffb2700e10) = 0 __sprintf_chk(0x7fffb2700fa0, 1, 4097, 0x407e89) = 23 __xstat(1, \u0026#34;/etc/cron.d/amavisd-new\u0026#34;, 0x7fffb2700e10) = 0 __lxstat(1, \u0026#34;/etc/crontab\u0026#34;, 0x7fffb2700cf0) = 0 open(\u0026#34;/etc/crontab\u0026#34;, 0, 00) = 4 __fxstat(1, 4, 0x7fffb2700cf0) = 0 After seeing the change, it uses a __lxstat and then the open function. The difference between normal functions and those prepended with two underscores, is that the latter are wrappers. Usually around the equally named function name, to provide a transparent wrapper and ensure the correct input and output.\nCouldn\u0026rsquo;t find .dynsym or .dynstr in \u0026hellip; Some files can not be traced with ltrace and may result something like:\nCouldn\u0026rsquo;t find .dynsym or .dynstr in \u0026ldquo;/proc/2098/exe\u0026rdquo;\nIf you encounter this error, you most likely have a statically linked binary. All functions are then inside the binary and not in a library. As ltrace can track libraries only, you have to use strace instead.\nList Open Files Everything on disk is a file. Normal files, devices and even directories are all presented as a file. The file system marks each of these entries in a file table, with the related type. While you might normally not even notice due to colored screen output, directories have a different type. It is the \u0026ldquo;d\u0026rdquo; in the first column in a long listing (-l) output of ls. Common types include a directory (d), a block or character based device (b/c) or a normal file (-, minus).\nTo see open files, we can use the lsof utility. It stands for \u0026ldquo;list open files\u0026rdquo; and definitely reveals its purpose. It can really show any type of open files, from the earlier mentioned special files (block and character devices), to tracking open network connections.\nOne big disadvantage of the lsof utility should be shared up front: there are way too many options to remember. The man page doesn\u0026rsquo;t make things easier, especially if you don\u0026rsquo;t know exactly what to look for. So getting exactly the right output is usually experimenting. At the bottom we have share some common used examples, to make this process easier.\nLsof commands Processes\nlsof -c cupsd = show open files for cups daemon Networking\nlsof -i -n = show open network connections (without name resolving) lsof -i4 or lsof -i6 = show IPv4 or IPv6 traffic lsof -N = show NFS Tracking syscalls with Linux Audit We already have written some posts about the powerful Linux audit framework. This built-in kernel feature allows tracking files and system calls. Of course we can combine both. We define what process we want to track and the related system call. Similar to strace we use the \u0026ldquo;open\u0026rdquo; system call.\nCreate rule: open Use auditctl to define a watch.\nauditctl -a always,exit -F arch=b64 -F pid=8175 -S open -k cups-open-files\nThis rule adds a system call monitor on \u0026ldquo;open\u0026rdquo; (with 64 bits architecture), for PID 8175. Now when this process uses the open system call, it will be logged in the audit log. We give it a key \u0026ldquo;cups-open-files\u0026rdquo;.\nSearch for file activity We can easily find them by referring to the earlier defined key \u0026ldquo;cups-open-files\u0026rdquo; with the ausearch command.\n# ausearch -k cups-open-files -- time-\u0026gt;Fri Jul 3 15:31:20 2015 type=CONFIG_CHANGE msg=audit(1435930280.293:390): auid=4294967295 ses=4294967295 op=\u0026#34;add rule\u0026#34; key=\u0026#34;cups-open-files\u0026#34; list=4 res=1 -- time-\u0026gt;Fri Jul 3 15:31:31 2015 type=PATH msg=audit(1435930291.853:392): item=0 name=\u0026#34;/etc/group\u0026#34; inode=5377472 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=CWD msg=audit(1435930291.853:392): cwd=\u0026#34;/\u0026#34; type=SYSCALL msg=audit(1435930291.853:392): arch=c000003e syscall=2 success=yes exit=4 a0=7fd5c0909351 a1=80000 a2=1b6 a3=0 items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=\u0026#34;cupsd\u0026#34; exe=\u0026#34;/usr/sbin/cupsd\u0026#34; key=\u0026#34;cups-open-files\u0026#34; -- time-\u0026gt;Fri Jul 3 15:31:31 2015 type=PATH msg=audit(1435930291.853:391): item=0 name=\u0026#34;/etc/passwd\u0026#34; inode=5377470 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=CWD msg=audit(1435930291.853:391): cwd=\u0026#34;/\u0026#34; type=SYSCALL msg=audit(1435930291.853:391): arch=c000003e syscall=2 success=yes exit=4 a0=7fd5c090935c a1=80000 a2=1b6 a3=0 items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=\u0026#34;cupsd\u0026#34; exe=\u0026#34;/usr/sbin/cupsd\u0026#34; key=\u0026#34;cups-open-files\u0026#34; -- time-\u0026gt;Fri Jul 3 15:31:31 2015 type=PATH msg=audit(1435930291.853:393): item=0 name=\u0026#34;/etc/cups/cups-files.conf\u0026#34; inode=5376875 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=CWD msg=audit(1435930291.853:393): cwd=\u0026#34;/\u0026#34; type=SYSCALL msg=audit(1435930291.853:393): arch=c000003e syscall=2 success=yes exit=4 a0=7fd5c55ce090 a1=0 a2=0 a3=23 items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=\u0026#34;cupsd\u0026#34; exe=\u0026#34;/usr/sbin/cupsd\u0026#34; key=\u0026#34;cups-open-files\u0026#34; -- time-\u0026gt;Fri Jul 3 15:31:31 2015 type=PATH msg=audit(1435930291.853:394): item=0 name=\u0026#34;/etc/group\u0026#34; inode=5377472 dev=08:05 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=CWD msg=audit(1435930291.853:394): cwd=\u0026#34;/\u0026#34; type=SYSCALL msg=audit(1435930291.853:394): arch=c000003e syscall=2 success=yes exit=5 a0=7fd5c0909351 a1=80000 a2=1b6 a3=2f items=1 ppid=1 pid=8175 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm=\u0026#34;cupsd\u0026#34; exe=\u0026#34;/usr/sbin/cupsd\u0026#34; key=\u0026#34;cups-open-files\u0026#34; -- The Linux audit framework is a great alternative to strace, but might be less friendly to configure. Especially on a system which already has watches going on, you might want to skip inserting a few test rules. In that case use strace instead.\nGot some others tools to track disk and file activity on running processes? Let it know!\n","permalink":"https://linux-audit.com/monitor-file-access-by-linux-processes/","tags":["auditctl","ausearch","ipv6","linux","lsof","strace","syscall"],"title":"Monitor file access by Linux processes"},{"categories":["Business","Career"],"contents":"How being a donkey can help you succeed in your daily work.\nThe donkey is considered to be a stupid animal in most of the western world. Actually, we might be pretty wrong about our current believes. As security professionals, we can definitely learn something from these friendly-looking creatures.\n1) Learn Quickly We have a Dutch saying \u0026ldquo;A donkey does not bump into the same stone twice\u0026rdquo;. The thought is that if you make the same mistake twice, you are stupid. If we reverse this though: the donkey is actually smart. It quickly learns from experiences and tries to avoid making mistakes over and over. This is one of the areas in which donkeys are much smarter than most animals, including humans.\nAs security professionals, we tend to be busy all day. We have to deal with a lot of information, resistance, and politics. On top of that, we have end-users which don\u0026rsquo;t want to comply with security policies, including system administrators, developers, and even upper management.\nLesson\nWhat we can learn from our furry friends is that we should adapt more quickly to our environment and learn from mistakes made in the past.\n2) Fight or Flight The donkey is a versatile animal, helping the human for centuries to transport, plowing land and actually protecting sheep against predators. It has a very logical way of thinking, in that it will fight or fly. It does not hesitate and makes quickly a decision on the current circumstances.\nAs security professionals, we are sometimes trapped between hidden agendas and different beliefs. Instead of clearly making a decision, we hide behind e-mail and policies.\nLesson\nThe behavior of the donkey is similar to the fight-or-flight mechanisms we humans had in the past. Nowadays we have too many choices,beliefs and internal company politics. Instead, we should quicker decide if we fight a situation or choose to step out, and let the person with the right authority make a decision.\n3) Weaknesses Donkeys have a weakness, which is the lack of being waterproof. This makes them vulnerable for rain and cold. The donkey knows this and needs a place to take shelter and keep itself protected against mother nature.\nAs security professionals, we have each our weaknesses. Unfortunately, most of the times we forget about this, especially when answering (difficult) questions from colleagues or customers. Too often we want to stay outside and answer everything, instead of taking shelter. It makes us vulnerable, possibly even annoying.\nLesson\nWhat we can learn from this example is that we should go back to a safe haven, especially when we don\u0026rsquo;t know. We also should involve other specialists, instead of battling alone.\nLike humans, the donkey has to learn by doing.\nConclusion Donkeys are not stupid, but a little bit stubborn and cautious (like us). Yet they can be very flexible, and take action when required. We should adopt their flexibility, to get our work done. Learn from questions, cases, and mistakes from the past.\nWe also should know our personal weaknesses, and accept help from others when needed. Last but not least, information security is about protecting assets and knowing risks. We should sooner make decisions and decide to fight, or flight. This way we ensure our efforts are helping the company and business. After all, security is not just a price tag to protect the business, but also to stay in business.\n","permalink":"https://linux-audit.com/three-lessons-for-security-professionals-be-a-donkey/","tags":["professional skills"],"title":"Three Lessons for Security Professionals: Be a Donkey!"},{"categories":["Auditing","Defensive Security","Hardening","Vulnerabilities"],"contents":"Every month or so, I get a few questions about the vulnerability capabilities Lynis has to offer. It made me think about this subject and I realized something: Many security professionals are still focusing too much on vulnerabilities. They want to know their security gaps, so they can know where they stand. While this isn\u0026rsquo;t a bad approach, there might be a better solution.\nThe solution I will discuss today is to focus on (permanent) processes, instead of vulnerability scanning. The goal is to reduce weaknesses quicker, and more often. Processes like software patch management, regular audits and security monitoring. So forget about vulnerability scanning and let\u0026rsquo;s proceed to the next level of security!\nVulnerability Scanning and Linux Systems When performing vulnerability scanning on Linux, it is common to find tools like Nessus and OpenVAS. These tools scan the network and optionally has the credentials to log onto systems and collect more information. The result is very binary: it discovered vulnerabilities, or it did not. We all know that seeing no single vulnerability is a rare exception.\nWhat Makes a System Weak? Vulnerabilities on a typical Linux system can be an outdated software package or a weak configuration. Buffer overflows are a common example of attacks to abuse vulnerabilities in the first category. The second category contains examples of two categories: lack of knowledge, or weakening on purpose. Let me clarify this one, as it is a root cause for many issues caused in past break-ins.\nToo often the configuration of a software package is weakened, by the system administrator (or developer!) making adjustments. Some setting may prevent you from getting your new application to work, so you turn it off. At the same time, this option was there to prevent specific attacks, with the result your system is now in a weakened state. Other examples for Linux include changing file permissions (chmod 777), turning of protection mechanisms (iptables, SELinux, AppArmor etc), or simply forgetting to remove default example files.\nThen there is the part of the knowledge gap. Still many system administrators don\u0026rsquo;t know how to configure or use the Linux audit framework. This is a shame, as it provides a very powerful way to do security monitoring. Vulnerability scanning is great, Linux security monitoring is even better.\nVulnerability Scanning is Negative Vulnerability scans provide a lot of findings. While this might look innocent at first sight, it also means that we always look at the negative side of security: vulnerabilities.\nVulnerabilities are by definition bad and a fact of life for system administrators. Unfortunately you can never win the game, as continuously new vulnerabilities are found. Keeping up with all the details and solving them, takes a lot of time. This is exactly most don\u0026rsquo;t have, as there are internal projects to finish and new systems to be deployed.\nWouldn\u0026rsquo;t it be better if we took a more positive approach when it comes to vulnerabilities? For example, we could look at the things we actually can do to increase the security defenses. These defenses could be the installation of a firewall, applying software patches and performing system hardening. In other words, we try to achieve continuous improvement. We make this part of our daily routine, until the leftovers are considered to be acceptable risks. It will be a long battle, but it is definitely possible to achieve a state of control.\nVulnerability management applied incorrectly Too often we see vulnerability scanning being done on environments, to show the need for more security. Similar to penetration testing of Linux systems. Both are actually steps which should be followed after the basics are properly in place. These basics contain of:\nSystem hardening Software patching Configuration file management Security monitoring Security audits Compliance testing Unfortunately, most companies don\u0026rsquo;t have these basic processes under control, yet use vulnerability scans and pentests to determine how well they are doing. The result is always bad\u0026hellip;\nContinuous Improvement The Japanese are considered to be the most skilled people in the world when it comes to quality. A great example is the story of Jiro Dreams of Sushi . In this story, Japanese sushi chef Jiro continues to keep improving his sushi servings. He isn\u0026rsquo;t the college graduate, but an 85-year old man who continues to keep learning. His believe that you should continuously strive to further improve. Perfection is something he doesn\u0026rsquo;t take for granted.\nTo know how to improve, you should know quality. We often take this word for granted. But what does quality actually mean? The core principle of quality is how well you can repeat something. If you create toothbrushes, you want them all to be like the original sample you created. So they should have the same strength, the same amount of brush hairs and the same color. If you can create an almost perfect copy of the original, quality is good.\nThis same practice of quality improvement is something we should consider more often in our field of expertise. Instead of focusing on the bad, we should look at the things that can be improved. We should not longer accept things as they are, but make informed decisions on what we can do to make small improvements and increase \u0026ldquo;IT quality\u0026rdquo;.\nWe can compare IT environments with a fabric: A lot of machinery, processes and humans involved. Instead of creating a physical product, we want to achieve some piece of output (e.g. keep the SaaS environment available for customers). We should learn from downtime, so we can decrease the chance and the impact of every negative event which occurs. At the same time we search proactively for improvements. This will result in quicker and more stable machinery, processes which are easier to understand and humans which know their roles and duties.\nIf we compare vulnerability scanning on Linux with our fabric example, we see that is might be a pointless exercise. It is like knowing that one machine is leaking oil, yet taking the old for granted. It is the actual action required to fix it, to improve: in this case changing a leaking bolt. So while vulnerability scanning is not a bad thing in itself, the focus should be on the routine checks. These checks could be done with regular audits. Software patch management is like regular maintenance. This way we can prevent a machine from leaking oil in the first place. When it still happens, we know that we can quickly discover it, to keep the impact to a minimum.\nAutomation on Linux Linux systems have a lot of great tools onboard to test security. These tools help with the continuous improvement process, by improving quality. Automation is a great tool to help with increasing quality.\nWith automation tools like Ansible, Chef and Puppet it is even easier than before to deploy small improvements. There is no need to download a hardening guide and do it all at once. Security should be part of your process and so every bit of hardening should be tested first, then deployed. Just pushing out a \u0026ldquo;full\u0026rdquo; policy, will sooner or later backfire.\nAvoiding backfire When pushing too many changes into production environments, we can actually harm the business. Still there are system administrators who rely too much on existing benchmarks and hardening guides, without proper testing. It might look innocent to change some kernel settings or software configurations. That is, until vague issues show up in the weeks after. Then it usually doesn\u0026rsquo;t take long that \u0026ldquo;security\u0026rdquo; got blamed and previous activities disabled. Good work backfires and results in even more work and a bad stance for security defenses. We can avoid this by taking the proper measures.\nSplitting work into small steps Instead of pushing big policies at once, the first thing to do, is splitting up work. Work can be done by category (e.g. all kernel settings), area (e.g. networking) or by priority and impact. The best strategy is dependent on your current stage and more interestingly on the level of monitoring. To avoid backfire, we should be able to implement changes and know their impact. For example increasing buffers to counter a denial-of-service attack, might increase memory usage. If the system has enough spare memory, this isn\u0026rsquo;t an issue. For a system being already challenged with the level of memory usage, this small adjustment might actually cause the denial-of-service!\nAlternatives for Linux vulnerability scanning Besides the focus on system hardening, we can focus on auditing and compliance. In this case, we measure the amount of systems which comply with the defined baseline. Also when using security measures, focus again on the positive. Sharing failure rates with your management will not positively impact your efforts. Instead, set a minimum baseline and a threshold to comply with. An example could the amount of systems which are part of the software patch management solution, with a minimum level of 95%. When 98% of your systems are subscribed to the central server, you know there is still some room for improvement, but also can show a positive trend.\nLinux security compliance When using baselines and hardening guides for your Linux environment, define what the minimum level should be. This could be a number or a percentage. This helps in defining when systems are compliant, or non-compliant. Even better is when linking it against your internal security policies and aligning security metrics.\nA few notes for security metrics. Make sure they can be measured properly. Ensure that the meaning behind a number or percentage is clear for your organization, to prevent misinterpretation. Like goals, the metrics should be achievable and realistic.\nExample security metrics\nPercentage of Linux systems being patched Percentage of systems audited last week Number of configuration files being managed by configuration management tools Conclusion Vulnerability management is actually a great tool to know your weaknesses. However, it does act from a negative standpoint, making it harder to sell the required action. The focus should be more positive, like well-defined processes (patching, hardening, auditing). Measuring and monitoring are key, to know where we stand and what next step to take. Again, Linux vulnerability management points out pain on your system, but shows data when it is already too late. Proactive improvements and regular maintenance are a better way to keep your Linux systems secure. Vulnerability management and penetration testing, should only be as a last level of validation.\nSo with these insights, forget about vulnerability scanning and focus first on the positive things that really matter.\n","permalink":"https://linux-audit.com/vulnerabilities/forget-linux-vulnerability-scanning-get-better-defenses/","tags":["software vulnerabilities","vulnerabilities","vulnerability management","vulnerability scan"],"title":"Forget Linux Vulnerability Scanning: Get Better Defenses"},{"categories":["Accounting","Auditing","Kernel Integrity","System Integrity"],"contents":"From Data and Logging, up to Kernel Integrity\nSystems exist for one primary goal, which is processing data. Information security helps protecting this valuable data, by ensuring its availability, integrity, and confidentiality. In other words, data should be available when we need it. Then it should be properly transmitted and stored, without errors. Our last goal ensures that it is only available to those with a need to know. Many open source software components are available to help with these goals. We will review a few of them and see how they fit in your security defenses.\nKnow what to protect You can\u0026rsquo;t protect something properly, if you don\u0026rsquo;t know it. Same applies for data and especially the type and underlying value of the data.\nSo data (or information) is valuable to you, or a business. However, what is valuable for you might be useless for someone else. At the same time something valuable to you, might be even more valuable to others. Think of your customer database, your financials and your strategies.\nSo before applying technical measures, get clear what it is that you are protecting.\nData types Some questions to help determine what you are protection, is first by determining what you have.\nCommon data types are health information, credit card details, personal information, contact details, trade secrets, public data, etc.\nWhat kind of data have you stored on the systems? Is there any sensitive data involved? Data storage Now next step is to determine where this data is stored. This helps later with selecting the right measures, depending on the data type.\nWhat systems (don\u0026rsquo;t share a name, just the type like: webserver) would have sensitive data? What systems have the most valuable data stored? Security is risk management After answering these questions above, the result could be a spreadsheet with multiple categories of data types and systems. One way to sort the sheet is by system. This should tell what types of data are stored on it. It helps with performing a risk assessment and focus first on those systems which handle more sensitive data.\nTime to check what measures we have and what kind of data it protects.\nData integrity If you process sensitive data, then the right combination of hardware and software should be used. For example, memory and disks have nowadays more reliable measures to ensure that all bits of data are correct. Any incorrect bit will be detected and reported. Then there are multiple levels of integrity mechanisms available within software.\nDatabases Database software can have atomic transactions, meaning that data should be committed to memory/disk only if all parameters are right. This helps greatly with making changes to data, while an relying piece of information was not stored (yet).\nDisk storage When we look at the disk itself, the hardware has features available to properly align between performance and security. Depending on how sensitive the data is, make sure systems and disks in particular have the time to perform a shutdown. This is especially valuable when there is a power outage.\nFile system integrity The last category worth mentioning is tooling which monitors file changes. This helps with general system integrity, by alerting when files have been changed. It is a great measure to detect intruders, but also ensure that changes to configuration files are properly documented. Unauthorized changes should be detected, so proper response actions can be taken.\nIn the same category there is the file system itself. When using newer file systems like EXT4, Btrfs and ZFS, options are available to guarantee the integrity of the data. Blocks which are damaged are early detected and disabled, to prevent malformed storage. They even can solve issues caused by the underlying disks, although that is to some extent.\nWhat measures to select? These measures are all worth using and are available and used on most systems already. At least you want to have the right disks and RAID level. On top of that the right memory modules, depending on the goal of the system. When highly sensitive data transactions occur, you might want to invest in memory modules with error detection (ECC).\nNext level of measures is on the storage. If data is important to you, it should be stored on the right type of storage. Select a stable storage solution, with the proper RAID level. It might be local disks, or network based storage. In any case, the value of the data should provide a guideline on what storage level is adequate.\nOn the file system level, pick the one which has the right performance, yet also protects data. Consult the details of the file system you are using and how to tune for it. If data needs more protection, check that the specific file system options are used, like disk journaling.\nOverview:\nCheck where data is stored Use the right hardware, optimized to the data processed and stored Apply settings of your file system Tune databases Kernel integrity The Linux kernel is the core of the operating system. It includes device drivers, system functions, memory management and much more low-level functions. To properly protect this core, we can take several measures.\nProtect the file system To protect the integrity of kernel, we should protect the areas where it is stored. A few important boot files, are usually stored in /boot.\nOptions:\nMount /boot read-only Monitor for write activities on /boot, /lib and user libraries Program integrity Besides the kernel itself, programs needs to be protected as well. Usually these are stored as binaries on disk. Depending on the permissions, users and processes can run a binary, which then perform a certain function.\nTo ensure only allowed binaries can be executed, the IMA/EVM system allows for denying unsigned binaries. This way malware and unauthorized binaries can no longer be executed. As this is an extensive subject, more posts about this will follow!\nAnother more common method is using file integrity tools. By monitoring changes to these files, we can quickly determine unauthorized changes. Another interesting thing to monitor is new binaries. They might indicate a normal installation, or the addition of a malicious file.\nFile integrity monitoring tools are both available as open source and commercial software. With tools like AIDE and Samhain, you have a great start.\nOptions:\nUse IMA/EVM for high sensitive systems Implement file integrity monitoring Log integrity Logging is the process of storing events in a certain way, for later access. It is used for debugging purposes, monitoring, accounting and forensic research. Too often these log files are taken for granted and not protected. It are these same files which help you discover what happened during a period, when something bad happened. Ensuring the integrity of these files, is therefore valuable for future events.\nTo protect log files, ensure that the file permissions and ownership are correct. Only the related daemon should be able to write to the file, to avoid unauthorized file alterations. Second level of ensuring integrity is to make files append only, and when possible store them remotely. This provides another barrier when a break-in on a system occurred.\n","permalink":"https://linux-audit.com/linux-system-integrity-explained-ensure-data-logging-and-kernel-integrity/","tags":["file integrity","file integrity monitoring","system integrity"],"title":"Linux System Integrity: Ensure Data, Logging and Kernel Integrity"},{"categories":["Malware","System Administration"],"contents":"Including the usage of Freshclam\nTo get ClamAV on CentOS installed, we have to use the EPEL repository (Extra Packages for Enterprise Linux). Fortunately, the Fedora project provides this with an easy installation. Unfortunately the default configuration is not properly working. In this post we collect some of the issues and required changes.\nLet\u0026rsquo;s start with installing the EPEL support.\nyum install epel-release\nNext step is installing all ClamAV components.\nyum install clamav-server clamav-data clamav-update clamav-filesystem clamav clamav-scanner-systemd clamav-devel clamav-lib clamav-server-systemd\nConfigure SELinux for ClamAV If you are using ClamAV on CentOS, together with SELinux, we should configure it a little bit. This way ClamAV can access all files on disk, and update its data definition files.\nEnable antivirus_can_scan_system:\nsetsebool -P antivirus_can_scan_system 1\nDuring database load : LibClamAV Warning: RWX mapping denied: Can\u0026#39;t allocate RWX Memory: Permission denied Configuration of Clam daemon Copy a the clamd.conf template, in case you don\u0026rsquo;t have a configuration file yet.\ncp /usr/share/clamav/template/clamd.conf /etc/clamd.d/clamd.conf sed -i \u0026#39;/^Example/d\u0026#39; /etc/clamd.d/clamd.conf Change /etc/clamd.d/clamd.conf file and define if you want to run the scanner as root, or a specific user. Check your /etc/passwd file for the related Clam user.\nChange the following two options:\nUser clamscan\nLocalSocket /var/run/clamd./clamd.sock\nEnable Freshclam Freshclam helps with keeping the database of ClamAV up-to-date. First delete the related \u0026ldquo;Example\u0026rdquo; line from /etc/freshclam.conf.\ncp /etc/freshclam.conf /etc/freshclam.conf.bak sed -i \u0026#39;/^Example/d\u0026#39; /etc/freshclam.conf Check the other options in the file, and change it to your preferred settings.\nMissing systemd service file We didn\u0026rsquo;t get a systemd service file, so creating a quick file here. The process should be forking itself and start freshclam in daemon mode. In this case we configure it to check 4 times a day for new files.\nCreate a new file /usr/lib/systemd/system/clam-freshclam.service\n# Run the freshclam as daemon [Unit] Description = freshclam scanner After = network.target [Service] Type = forking ExecStart = /usr/bin/freshclam -d -c 4 Restart = on-failure PrivateTmp = true [Install] WantedBy=multi-user.target Now enable and start the service.\nsystemctl enable clam-freshclam.service systemctl start clam-freshclam.service Check the status.\n# systemctl status clam-freshclam.service clam-freshclam.service - freshclam scanner Loaded: loaded (/usr/lib/systemd/system/clam-freshclam.service; enabled) Active: active (running) since Thu 2015-06-11 11:09:24 CEST; 1s ago Process: 3158 ExecStart=/usr/bin/freshclam -d -c 4 (code=exited, status=0/SUCCESS) Main PID: 3159 (freshclam) CGroup: /system.slice/clam-freshclam.service └─3159 /usr/bin/freshclam -d -c 4 Change service files By default, the service files seem to be messy and not working.\nThese are the files bundled:\n# ls -l /usr/lib/systemd/system/clam* -rw-r--r--. 1 root root 136 Apr 29 20:38 /usr/lib/systemd/system/clamd@scan.service -rw-r--r--. 1 root root 231 Apr 29 20:38 /usr/lib/systemd/system/clamd@.service When enabling the clamd service, we would see something like this:\n# systemctl enable /usr/lib/systemd/system/clamd@.service Failed to issue method call: Unit /usr/lib/systemd/system/clamd@.service does not exist. So let\u0026rsquo;s fix it. First rename the /usr/lib/systemd/system/clamd@.service file.\nRename the clamd@ file.\nmv /usr/lib/systemd/system/clamd@.service /usr/lib/systemd/system/clamd.service\nNow we have to change the clamd@scan service as well, as it refers to a non-existing file now. Change this line in /usr/lib/systemd/system/clamd@scan.service and remove the @ sign.\n.include /lib/systemd/system/clamd@.service\nNext step is changing the clamd service file /usr/lib/systemd/system/clamd.service\n[Unit] Description = clamd scanner daemon After = syslog.target nss-lookup.target network.target [Service] Type = simple ExecStart = /usr/sbin/clamd -c /etc/clamd.d/clamd.conf --foreground=yes Restart = on-failure PrivateTmp = true [Install] WantedBy=multi-user.target Move into the directory.\ncd /usr/lib/systemd/system\nStart all services.\n# systemctl enable clamd.service # systemctl enable clamd@scan.service # systemctl start clamd.service # systemctl start clamd@scan.service Checking the status With all these changes, ClamAV on CentOS 7 should be running now. The easiest way to check, is using the ps command and see if freshclam and clamd are running.\nUseful resources for debugging are the systemctl status command, followed by the service. Then there is logging in /var/log/messages, which usually will reveal when and why something is (not) running.\nMore tips? Let it know!\n","permalink":"https://linux-audit.com/install-clamav-on-centos-7-using-freshclam/","tags":["clamav"],"title":"Installing ClamAV on CentOS 7 and Using Freshclam"},{"categories":["System Administration"],"contents":"Proper software patch management helps reducing weaknesses on your systems. But even if you patched an outdated system, old processes and libraries can continue to run in memory. For example when a library is updated, an active program might still use the old version. To really finish the process of software patching, we have to do more. This includes preparation, performing the update and finally check if we need a restart of software components. In this post we have a look at several options, to properly execute this last part of the process. An introduction into the world of tools like checkrestart and needrestart.\nCheckrestart The first utility to help with the job of finding processes using old files, is checkrestart. It is part of the debian-goodies package and only available for Debian based systems. It uses lsof to determine open files and what processes using such resource.\nInstallation apt install debian-goodies\nRequirements Debian (or clone) Python LSOF root permissions Usage Running the checkrestart command will give an overview of what it discovered and what processes need a restart. It shows the processes using old files and determines what init scripts are related to these processes. Of course, those which it can find. For the others it will display the related processes, so you can manually take action.\nExample output # checkrestart lsof: WARNING: can\u0026#39;t stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs Output information may be incomplete. Found 68 processes using old versions of upgraded files (48 distinct programs) (40 distinct packages) Of these, 7 seem to contain init scripts which can be used to restart them: The following packages seem to have init scripts that could be used to restart them: sudo: 3908 /usr/bin/sudo cups-browsed: 1457 /usr/sbin/cups-browsed samba: 965 /usr/sbin/smbd 700 /usr/sbin/smbd 2371 /usr/sbin/nmbd mdm: 1491 /usr/sbin/mdm pulseaudio: 3039 /usr/bin/pulseaudio cups-daemon: 3568 /usr/sbin/cupsd dbus: 2822 /bin/dbus-daemon 3477 /bin/dbus-daemon 2385 /bin/dbus-daemon 2836 /bin/dbus-daemon 507 /bin/dbus-daemon These are the init scripts: service sudo restart service cups-browsed restart service samba-ad-dc restart service smbd restart service samba restart service nmbd restart service mdm restart service pulseaudio restart service cups restart service dbus restart These processes do not seem to have an associated init script to restart them: udisks2: 3162 /usr/lib/udisks2/udisksd policykit-1: 918 /usr/lib/policykit-1/polkitd modemmanager: 783 /usr/sbin/ModemManager blueman: 3255 /usr/bin/blueman-applet network-manager: 912 /usr/sbin/NetworkManager mate-polkit:amd64: 3257 /usr/lib/x86_64-linux-gnu/polkit-mate-authentication-agent-1 xserver-xorg-core: 1498 /usr/bin/Xorg gvfs-daemons: 3143 /usr/lib/gvfs/gvfs-udisks2-volume-monitor 3554 /usr/lib/gvfs/gvfsd-metadata 3321 /usr/lib/gvfs/gvfsd-trash mate-keyring: 2996 /usr/bin/mate-keyring-daemon caja: 3236 /usr/bin/caja system-tools-backends: 3487 /usr/sbin/system-tools-backends gvfs-backends: 3214 /usr/lib/gvfs/gvfs-mtp-volume-monitor 3203 /usr/lib/gvfs/gvfs-afc-volume-monitor 3208 /usr/lib/gvfs/gvfs-gphoto2-volume-monitor marco: 3015 /usr/bin/marco upower: 3272 /usr/lib/upower/upowerd mintupdate: 3821 /usr/lib/linuxmint/mintUpdate/mintUpdate.py mate-screensaver: 3258 /usr/bin/mate-screensaver at-spi2-core: 3472 /usr/lib/at-spi2-core/at-spi-bus-launcher dnsmasq-base: 2142 /usr/sbin/dnsmasq 3543 /usr/sbin/dnsmasq mate-bluetooth: 3261 /usr/bin/mate-bluetooth-applet consolekit: 2476 /usr/sbin/console-kit-daemon gvfs-fuse: 3023 /usr/lib/gvfs/gvfsd-fuse google-chrome-stable: 3646 /opt/google/chrome/chrome 3859 /opt/google/chrome/chrome 3642 /opt/google/chrome/chrome 3683 /opt/google/chrome/chrome 3675 /opt/google/chrome/chrome 3666 /opt/google/chrome/chrome 3612 /opt/google/chrome/chrome 3691 /opt/google/chrome/chrome 5706 /opt/google/chrome/chrome 3694 /opt/google/chrome/chrome 3601 /opt/google/chrome/chrome 3698 /opt/google/chrome/chrome 3621 /opt/google/chrome/chrome 3708 /opt/google/chrome/chrome 3616 /opt/google/chrome/nacl_helper mate-terminal: 3891 /usr/bin/mate-terminal mate-settings-daemon-pulse: 2987 /usr/bin/mate-settings-daemon system-config-printer-gnome: 3252 /usr/share/system-config-printer/applet.py mate-session-manager: 2547 /usr/bin/mate-session isc-dhcp-client: 2024 /sbin/dhclient mintmenu: 3049 /usr/lib/linuxmint/mintMenu/mintMenu.py mate-media-pulse: 3259 /usr/bin/mate-volume-control-applet network-manager-gnome: 3248 /usr/bin/nm-applet 3263 /usr/bin/nm-applet mate-panel: 3055 /usr/lib/mate-panel/clock-applet 3032 /usr/bin/mate-panel 3051 /usr/lib/mate-panel/wnck-applet login: 3909 /bin/su mate-power-manager: 3244 /usr/bin/mate-power-manager From this output we can see what processes can be restarted, along the related init script. Additionally it shows what processes need also restarting, but lack the related script.\nCheckrestart status The utility does what it should do. However, it is limited to Debian based systems and the code contains a lot of \u0026ldquo;to do\u0026rdquo; items. If you are evaluating software components in this article, you definitely want to read more about Needrestart.\nNeedrestart Another utility is needrestart. Like checkrestart it determines what processes need a restart, after running a software upgrade. It is part of the package with the same name and written in Perl. It seems to be well-maintained and supports newer technologies like containers (LXC, Docker). The tool hooks into the update process, so for example it can restart services after running a \u0026ldquo;dpkg upgrade\u0026rdquo;. It is also possible to run it manually, with specifically the mode to list processes only, provide the option to do restarting (interactively) per process, or do them all automatically.\nSupport for package managers Needrestart supports a few package managers out of the box:\nDPKG Pacman RPM Needrestart restart uses features of the package manager to determine which related package, or daemon needs a restart. It does so by looking for the related startup script. In the case of systems using DPKG, it actually uses some intelligence from the previously mentioned checkrestart utility. For RPM it leverages the rpmquery utility, for pacman the pacman utility itself.\nRunning security updates, before needrestart comes into play\nInstallation Debian / Ubuntu apt install needrestart\nFedora For our Fedora system we used the following steps to get the tool working. Instead of using Git, you might use a custom package to simplify the handling of dependencies. As we simply use it for testing on our Fedora 21 test server, we install the dependencies and run it from the root home directory.\n# cd /root # yum -y -d1 install git perl-Module-ScanDeps perl-Proc-ProcessTable perl-Sort-Naturally perl-Term-ProgressBar-Simple perl-Module-Find.noarch perl-ExtUtils-MakeMaker.noarch # git clone https://github.com/liske/needrestart # cp /root/needrestart/needrestart.conf /etc/needrestart/needrestart.conf # mkdir /etc/needrestart/hook.d # cp /root/needrestart/ex/hooks/* /etc/needrestart/hook.d # perl -I /root/needrestart/perl/lib ./needrestart -r l\u0026lt;/pre\u0026gt; Configuration of Needrestart By default, not much configuration is needed for this utility. It works great out of the box. When adjustments are needed for the behavior of the tool, this can be done via the configuration file /etc/needrestart/needrestart.conf . Some of options that are found in the configuration file, can also be adjusted via the command line (e.g. type of operation).\nSo what is there to configure? The default restarting mode, what processes should be ignored etc. This way you can change the utility to do its job, while avoiding restarting unwanted parts of the system.\nNeedrestart in action So after running security updates, we will run needrestart. In this example, we use it on a Fedora system. Just after applying hundreds of software patches, we run the tool in list mode (-r l):\nKernel and software restart needed\nlinux-audit-needrestart-kernel-software-restart-needed.png\nNeedrestart found several processes which needed a restart. The tool even checks if the latest installed kernel is running and discovered an outdated kernel being active. In this case a full system restart would even be better.\nWhich one to use? After reviewing them both, go for the needrestart utility. It works on multiple Linux based systems, is well-maintained, has support for newer technologies and does it job very well. As always, we encourage testing first on non-production systems.\nFound even a better tool to handle this job? Let it know!\n","permalink":"https://linux-audit.com/determine-processes-which-need-a-restart-with-checkrestart-needrestart/","tags":["debian","dpkg","fedora","patch management","rpm","software patching"],"title":"Determine which processes need a restart with checkrestart/needrestart"},{"categories":["Business"],"contents":"How to Get Money for a New Security Tool\nWe all know the common answer when asking for a new software tool: \u0026ldquo;sorry, no budget\u0026rdquo;. But why is that? Often because we, as technical oriented people, simply don\u0026rsquo;t know how budgeting works. Not surprising, as no one taught us. The downside is that it limits us seriously, to obtain the right tools for the job. Time to combine tech, money, and skills, to get finally that new tool you wanted!\nWhy budgeting? Most bigger companies have a formal budgeting process in place. It is actually a security measure, to ensure that financial risks are kept under control. And guess what, this is also why there are \u0026ldquo;controllers\u0026rdquo;, people who manage the budgets. Besides them, a big company may also have a Chief Financial Officer, or CFO, responsible for the financials of the company.\nOperations On the operational level, the budgets are usually managed by IT managers. They have actually a difficult challenge. Their job is to ensure that the IT environment is running stable, while avoiding money draining too quickly. This brings us to the next point, forecasting.\nBudget forecasting Let\u0026rsquo;s say you are a manager and responsible for a team of engineers, hundreds of servers, software licenses, and a data center. You will need a fair amount of money every year to keep it running, right? So after a few years, you did some calculations and know exactly what is needed. Unfortunately, it is not that easy.\nAnother bad surprise Even with rigorous budgeting effort, unwelcome surprises may show up in the mailbox of the manager. One of the best engineers might be leaving, or an internal department suddenly needs 10 systems to host their new application. Of course, something they didn\u0026rsquo;t share a month ago while talking with them. Whatever the surprise might be, it usually costs money. Money which was not part of the initial budget.\nWhen you ask for a new tool, you might be actually part of this problem of surprises. If you want to make your manager happy and get more things arranged, be part of the solution, not the (budget) problem.\nBudget cycle Budget cycles are usually defined for a fixed period. This period mainly depends on how the company is structured. Some do it every quarter, others twice a year, or just once a year.It is common to see that the budget round finishes a few months before the\nIt is common to see that the budget round finishes a few months before the end of the year or the end of the financial year. Again, it depends on the company and how the financials are arranged. The budget cycle simple is adjusted. While this may be a simple fact, it is crucial for obtaining budget, as we will see later.\nBuffers in budgets To counter unexpected events, usually budgets are stretched beyond what is really needed. Also items which higher risk, like that unstable database cluster, might be included as a replacement cost. Besides the unexpected events, the manager may also include expected growth. If your server farm grows each year with 5%, you know that this won\u0026rsquo;t be equally divided over the year. So depending on the budget cycle, it may be better to ask more, than asking too little. This is especially true if the manager needs to explain for the 5th time why he has a budget overrun.\nObtaining budget So you want to have a new tool in your daily activities, or as an extension of your toolkit. From previous experiences you get the \u0026ldquo;no budget\u0026rdquo; answer, or others tell you it won\u0026rsquo;t happen.\nFor the sake of providing an example, let\u0026rsquo;s use our tool Lynis. You discovered the free open source version, and totally got in love with it. After using it for a few months, you learn about the enterprise version and want to have that in your toolkit. To avoid the budget issue, you might ask yourself how to get around this. Let\u0026rsquo;s craft a budget plan!\nCrafting the plan Good plans are executed in steps. So let\u0026rsquo;s divide our master plan into 3 big steps:\nValue Timing Proof Step 1: Budget equals value Budget is about money. Money is about trading. Trading is about exchanging value. So in other words, budget is about value. This also applies to software tools, open source, closed source, or the mix. In all cases, tools provide some level of value, but it may differ per user of the software.\nDetermine Personal Value First step is to know what the value of the tool is for your organization. Don\u0026rsquo;t see it as another great toy, but really think about how it helps you in your daily work. Does it save you time? Does it remove tedious repeating steps?\nExample: Lynis saves you time by automation. Checklists can be more often checked, and at the same time quicker and more reliable.\nDetermine Team Value Then make it broader and see how the tool value relates to your manager and the team. Does it save money in the short or long term? Does it improve the quality of the IT environment? Does it make your team look more experienced?\nExample: By using an auditing tool, we can reduce inconsistencies in our environment, which otherwise may lead to disruptions. We can find vulnerabilities which normally would stay undetected and may result in data disclosure.\nSell the Value Keep in mind that you will have to \u0026ldquo;sell\u0026rdquo; the solution to your manager. This way he/she is able to reserve some budget for it. At the same time, the manager of the manager needs to be convinced as well. So if you write a proposal, don\u0026rsquo;t write it just in terms for your manager, but make it easy to forward. If you want a new tool, it is time to start selling!\nOne strong characteristic when selling your value proposition to your manager is providing examples. Show that you have done your diligence when searching for a solution. Explain how this solution can help. Don\u0026rsquo;t be afraid to show your weaknesses. We like to cover up things we discovered, in the hope that your manager never finds out. But it is better to show him you are doing a great job, but want to ensure there is an additional set of (cheap) eyes, testing it on a daily basis. From auditing tools up to vulnerability scanners, they help us to discover the things we simply couldn\u0026rsquo;t find ourselves, due to knowledge or time constraints.\nExample: Currently we have to use custom built scripts to check if security patches are needed. This solution covers that. It goes much further, including checking other areas which are not part of our monitoring tooling. The free version already discovered 5 systems which had an outdated time server, something which is normally set during installation and not tested later on. With this tool we can more easily detect these and fix them, before they result in service disruption.\nUse the Vendor The vendor of a software solution knows the difficulties with budgets. As a founder, I\u0026rsquo;ve seen it myself. So tell in an early stage how your company deals with budgets. Also, ask if they can help with getting the value more clear to your company. Most of us techies are too careful to share details, thinking that the evil sales person may abuse this information.\nWhile it is good to avoid sharing confidential data, giving insights on how your business operates, is valuable for both. It helps the vendor to understand your needs better and in some cases adjust the solution, to provide a better value for the money. Remember, it is all about the value and selling it internal.\nStep 2: Choose the Right Moment Even if have the best proposal ever, timing can be a nasty deal-breaker. So to counter this, you have to learn a little bit about the budgeting cycle of your company. This understanding may also help your manager. Tell him/her that you understand that budgeting can be challenging. Then ask what the soft and hard deadline is of the next cycle. The hard deadline is when everything has to be approved. However, it is the soft deadline which is interesting for you. That is the moment when your manager needs to submit his budget forecast.\nStep 3: Show Proof That it Works Knowing and selling the value of a solution is great. Perfect timing makes it even better. But the real winner is to lock the budget for the years to come. After all, you might need to pay a yearly fee. Other examples include upgrading the support contract to the next level, or simply the need for more licenses with a continuous growing server fleet.\nProof, Proof, and Proof We all know that happy feeling when purchasing something new. But how often do we look back and consider the purchase to be a great investment? Even if you feel great with a particular software solution, you manager might think otherwise. To be sure your tool gets through another budget round, we need those gentle reminders that the solution is good. For that our first two steps come into play, again.\nRepeat the Value One method to show proof that a solution is working, is by using dashboard functionality. Show on a screen that everything is \u0026ldquo;green\u0026rdquo;, because you are using this tool. Optionally, use reports to show the discoveries your tool made. Even better is when it can show how it helped improving the quality of your environment. Maybe the amount of IT incidents dropped hugely, because of otherwise repeating issues.\nTiming Like the initial budget request, timing remains important. Your next budget requests will depend on them. Also when showing proof in the form of value to the company, timing can make a big difference. Ensure that it is done on a regular basis, so there is no doubt when another round of budget savings is announced.\nOpen source, Security and Money Those who deal a lot with security in their job, are usually a lot of open source components. While free tools are great, we might perceive that security can be cheap by limiting ourselves to open source only. Even as an open source developer myself, I like to warn you about the bad consequences it may have for your company. For example when we look at the GNU/Linux kernel itself. Great that we can use it for free, but that is because of many commercial companies providing support, doing marketing and more importantly, submit patches. Most of the development and commits are nowadays done by big companies like Intel and Red Hat.\nNo Budget = Limited Growth If you simply focus on cheap solutions, you will end up with limited resources. Open source, and especially free software, is great. But if you don\u0026rsquo;t have any budget to get a proper toolkit in place, you won\u0026rsquo;t get the maximum benefit out of your team.\nPeople are Expensive, Tools are Not Consider the cost of your company to hire you, every single month. Now multiply that by 12 and compare it with license of 1 single tool. Now is that 1200 euro/dollar really that much? Don\u0026rsquo;t think so!\nSecurity is a Hard Sell Security is a strange thing. Some people compare it with insurance. It is ok when you don\u0026rsquo;t have it. When something happens, then you wished you did have it. The biggest problem is explaining the value of a solution.\nWhen looking at security products, have a look at how it helps reducing risks. Try to make it as tangible as possible. If you express something in a number, then do so. A great example is when you are using a CMDB and change/incident management tool. Have a quick look at the tickets in the last year and determine what happens often. It may provide a number on what kind of issues may be reduced, after implementing the new solution.\nExample: by using an auditing tool we can measure how effective our policies are. We can then calculate the amount of systems being non-compliant and focus our time on the machines with missing security measures. Additionally, we can create reports for our customers, to help them deploying their applications more securely. In the last 6 months, we had at least 12 machines with a security breach, due to weak configurations, which otherwise would be detected up-front.\nPractical tips If you made it till the end of this blog post, it is time to receive an additional bonus. After all, speaking about value and timing, this is the spot to make it even better. Here are some practical tips which help to understand budgets and achieve your goals.\nLearn from your manager about budgeting Set a reminder in your agenda in advance of the budget cycle Discuss with your team what tools are needed and make a top 3 sorted by value Search or create an internal budget request template, to know what information is needed for approval Involve your manager, and get easier buy-in for the new purchase Make use of the vendor to supply enough information, and reduce purchasing risks Got more tips for others who struggle to get budget? Let it know and we can add it to the article.\n","permalink":"https://linux-audit.com/budgeting-for-techies-how-to-get-money-for-a-new-security-tool/","tags":["tips","tools"],"title":"Budgeting for Techies: How to Get Money for a New Security Tool"},{"categories":["Kernel Integrity"],"contents":"The Linux kernel is modular, which makes it more flexible than monolithic kernels. New functionality can be easily added to a run kernel, by loading the related module. While that is great, it can also be misused. You can think of loading malicious modules (e.g. rootkits), or unauthorized access to the server and copy data via a USB port. In our previous article about kernel modules, we looked at how to prevent loading any module. In this case, we specifically disallow the ones we don\u0026rsquo;t want.\nBlacklisting modules Blacklisting modules is one way to disallow them. This defines which modules should no longer be loaded. However, it will only limit the loading of modules during the boot process. You can still load a module manually after booting.\nBlacklisting a module is simple. Create a file in the /etc/modprobe.d directory and give it a proper name (e.g. blacklist-module.conf).\nBlacklisting firewire Let\u0026rsquo;s say we want to blacklist firewire. We first have to determine what modules are available. By using find, we can quickly determine the related kernel drivers:\n# find /lib/modules/`uname -r` -name *firewire* /lib/modules/4.0.1-1-ARCH/kernel/drivers/firewire /lib/modules/4.0.1-1-ARCH/kernel/drivers/firewire/firewire-ohci.ko.gz /lib/modules/4.0.1-1-ARCH/kernel/drivers/firewire/firewire-core.ko.gz /lib/modules/4.0.1-1-ARCH/kernel/drivers/firewire/firewire-sbp2.ko.gz /lib/modules/4.0.1-1-ARCH/kernel/drivers/firewire/firewire-net.ko.gz /lib/modules/4.0.1-1-ARCH/kernel/drivers/media/firewire /lib/modules/4.0.1-1-ARCH/kernel/drivers/staging/fwserial/firewire-serial.ko.gz /lib/modules/4.0.1-1-ARCH/kernel/sound/firewire /lib/modules/4.0.1-1-ARCH/kernel/sound/firewire/snd-firewire-lib.ko.gz Now we know there are multiple modules, most part of the drivers and one in the sound section. If we want to disable all these modules, we could simply blacklist them all. Or block the generic category.\nGathering module information By using modinfo, we can gather the details about a particular module. In this case, we have a look at the snd-firewire-lib module and see what it does:\nThe modinfo commands shows on which a module depends\nWe can see it depends on firewire-core. Let\u0026rsquo;s have a look at the firewire-core module itself:\nDetails of firewire core module\nThe details of the firewire-core module show that is responsible for firewire itself. It is the core unit itself and doing the transaction logic within the IEEE1394 protocol specifications. We can see it is depending on the CRC-ITU-T standard.\nBy blacklisting the firewire-core, we effectively disable any module depending on it. In this case, we don\u0026rsquo;t blacklist the crc-itu-t module, to prevent other modules from properly functioning.\nThe related snippet to blacklist would be:\n/etc/modprobe.d/blacklist-firewire.conf\nblacklist firewire-core\nSee blacklisted modules To see what modules are currently blacklisted, we can use the modprobe command:\n# modprobe --showconfig | grep blacklist blacklist firewire_core This will show all modules which are blacklisted.\nDisable modules The next level of blacklisting modules is to actually disable them. This way they won\u0026rsquo;t be loaded unintentionally.\nTo disable a module, we have to redirect a module via the install option. Modprobe will try to load the related file. By defining a module as /bin/true, it won\u0026rsquo;t be loaded.\nUsing the install option we can avoid loading modules\nTo see what modules are currently disabled via install, we can use modprobe as well:\n# modprobe --showconfig | grep \u0026#34;^install\u0026#34; | grep \u0026#34;/bin\u0026#34; install firewire_core /bin/true install firewire_ohci /bin/true Note: the root user can still override settings, by using the --ignore-install parameter. In that case, the module can still be loaded.\nConclusion By using the right combination of blacklist, install and alias, we can disallow the loading of Linux kernel modules. They form the first level of defense against unintentional and unauthorized module loading. By using the kernel setting kernel.modules_disabled and set its value to 1, we can make sure things are really tightened. Even the root user can not load any modules anymore.\nUseful commands When working with kernel modules, here are some of the most common commands:\nBlacklisted and disabled modules modprobe -showconfig | egrep \u0026ldquo;^(blacklist|install)\u0026rdquo; Find modules find /lib/modules/`uname -r` -print Show loaded modules lsmod Load module modprobe module Unload module modprobe -r module Module details modinfo module Questions or other tips? Let it know!\n","permalink":"https://linux-audit.com/kernel/kernel-hardening-disable-and-blacklist-linux-modules/","tags":["kernel","kernel modules","linux","lsmod","usb"],"title":"Kernel hardening: Disable and blacklist Linux modules"},{"categories":["Hardening","Kernel Integrity","Linux"],"contents":"Disable loading kernel module on Linux systems\nThe Linux kernel can be configured to disallow loading new kernel modules. This feature is especially useful for high secure systems, or if you care about securing your system to the fullest. In this article, we will have a look at the configuration of this option. At the same time allowing legitimate kernel modules to be loaded.\nDisable kernel modules Newer kernel modules have a sysctl variable named kernel.modules_disabled.\nSysctl is the tool which allows you to see and change kernel settings of a running system. The related /etc/sysctl.conf file is used to ensure that your settings are also used at the next boot of the system.\nThe sysctl key kernel.modules_disabled is very straightforward. If it contains a \u0026ldquo;1\u0026rdquo; it will disable loading new modules, where a \u0026ldquo;0\u0026rdquo; will still allow loading them.\nUsing this option will be a great protection against loading malicious kernel modules. For example, it may help to counter rootkits. Needless to say, but when someone was already been able to gain root access, you have a serious problem. Still, setting this security measure can be useful to achieve maximum hardening of your Linux system. An altered script or program has no chance of loading things you didn\u0026rsquo;t specifically approve.\nLoading modules To show this functionality, we first will load a module and then see how the related sysctl value works. For this demo purpose, we will use the XOR module, which is part of the crypto category.\n# cd /lib/modules/3.13.0-24-generic/kernel/crypto/ # ls -l total 1012 -rw-r--r-- 1 root root 8516 may 3 2014 ablk_helper.ko -rw-r--r-- 1 root root 20436 may 3 2014 af_alg.ko -rw-r--r-- 1 root root 13892 may 3 2014 algif_hash.ko -rw-r--r-- 1 root root 17748 may 3 2014 algif_skcipher.ko -rw-r--r-- 1 root root 11772 may 3 2014 ansi_cprng.ko -rw-r--r-- 1 root root 15756 may 3 2014 anubis.ko -rw-r--r-- 1 root root 6828 may 3 2014 arc4.ko -rw-r--r-- 1 root root 17180 may 3 2014 authencesn.ko ..snip.. -rw-r--r-- 1 root root 26076 may 3 2014 twofish_common.ko -rw-r--r-- 1 root root 10028 may 3 2014 twofish_generic.ko -rw-r--r-- 1 root root 13540 may 3 2014 vmac.ko -rw-r--r-- 1 root root 31372 may 3 2014 wp512.ko -rw-r--r-- 1 root root 9028 may 3 2014 xcbc.ko -rw-r--r-- 1 root root 21124 may 3 2014 xor.ko -rw-r--r-- 1 root root 10596 may 3 2014 xts.ko -rw-r--r-- 1 root root 16396 may 3 2014 zlib.ko Background information The XOR kernel module uses the \u0026ldquo;exclusive OR\u0026rdquo; function, which returns True when an odd number of the given arguments equals to be true, and False when an even are true. Usually it is used with two arguments, so to return True, only 1 of the options can be positive.\nBack to the loading of the module. First we ensure this module is not loaded:\nlsmod | grep xor\nNow we load the module with the insmod command and then check if loading was successful:\n# insmod xor.ko # lsmod | grep xor xor 21411 0 The module is loaded, as expected. To show normal behavior, we now will remove the kernel module with the rmmod command.\n# rmmod xor # lsmod | grep xor The module is released. Time to disable this functionality, to increase protection against loading malicious modules.\nActivating kernel.modules_disabled By default, the sysctl key is set to \u0026ldquo;0\u0026rdquo;, which means new modules can be loaded. This is a safe default for systems but also allows malicious modules to be loaded.\n# sysctl -a | grep modules kernel.modules_disabled = 0 Now we disable loading new modules, by using the sysctl key and set it to \u0026ldquo;1\u0026rdquo;. There are two ways of doing it, using sysctl directly or echo the value to a file on the pseudo file system /proc, which holds the kernel settings.\necho 1 \u0026gt; /proc/sys/kernel/modules_disabled\nNow we try loading our XOR module again:\n# insmod xor.ko insmod: ERROR: could not insert module xor.ko: Operation not permitted Loading the module is now no longer allowed, exactly what we wanted.\nProtection against re-enabling You might think that loading a kernel module is as simple as re-enabling the option and then still load your kernel module. The kernel has a built-in protection, to avoid this from happening. Trying to set the value back to \u0026ldquo;0\u0026rdquo; will result in an \u0026ldquo;invalid argument\u0026rdquo; message.\nSysctl showing invalid argument when trying to set value\nAs can be seen, sysctl will say the value is set to \u0026ldquo;0\u0026rdquo;. However, the value isn\u0026rsquo;t applied, as this key is read-only. Slightly confusing, and therefore always good to check the value again.\n# sysctl kernel.modules_disabled kernel.modules_disabled = 1 As expected, the value is still set to \u0026ldquo;1\u0026rdquo;.\nDisable module loading after boot time By configuring the /etc/sysctl.conf file we can disallow the loading of kernel modules at boot time. Simply add the related line, with the value \u0026ldquo;1\u0026rdquo; as shown in the example.\nCaveat: Things might break Depending on your environment, you might be careful with using this option. It may be working very well on servers, but not on desktop systems. The reason is the type of usage is different, especially when it comes with loading new kernel modules. For example inserting a USB drive, mouse or network functionality might break. So before deploying the option, make sure you test these common use cases.\nHybrid option Instead of enabling the option directly via /etc/sysctl.conf, it might be better to activate this setting after booting and loading required modules.\nYour startup script could be looking like:\n#!/bin/sh/ sleep 300 insmod MODULENAME echo 1 \u0026gt; /proc/sys/kernel/modules_disabled Usually to get iptables working, these are the related modules: iptables, x_tables, iptable_filter.\nDepending on your Linux distribution, the startup should be loaded as late as possible. If you have /etc/rc.local available, that is usually a safe bet.\nDo you use this option already? Or found some other caveats? Like to hear!\n","permalink":"https://linux-audit.com/kernel/increase-kernel-integrity-with-disabled-linux-kernel-modules-loading/","tags":["kernel","linux","lsmod"],"title":"Increase kernel integrity with disabled Linux kernel modules loading"},{"categories":["Accounting","Auditing"],"contents":"Capturing execve system calls and store them in the audit log\nFor compliance or security reasons you might want to capture all commands executed by the root user. Fortunately enough the Linux Audit Framework helps with capturing the right system calls and log it to the audit file.\nConfigure audit To enable auditing, use the following two auditctl commands:\n# auditctl -a exit,always -F arch=b64 -F euid=0 -S execve -k root-commands # auditctl -a exit,always -F arch=b32 -F euid=0 -S execve -k root-commands These commands will enable monitoring for the execve(2) system call (32-bit and 64-bit), but only when the effective user ID is 0, equal to the root user. Whenever you are logged in as root, or using sudo, it will log the related actions.\nAn alternative method is capturing all system calls for the root user, with a permission based filter. This means that all executes of files, write actions and changes to attributes are recorded.\nauditctl -a exit,always -S all -F euid=0 -F perm=awx -k root-commands\nSearching root activities After logging the events for a while, we might want to search for them. By specifying a key (root-commands), you can quickly find them again using the ausearch command.\nausearch -k root-commands\nThis will perform a search through all audit entries, for which the key is root-commands.\nAnother option to capture administrative commands, is using Snoopy. Have a look at our previous blog post about Snoopy.\n","permalink":"https://linux-audit.com/logging-root-actions-by-capturing-execve-system-calls/","tags":["audit","auditctl"],"title":"Logging root actions by capturing execve system calls"},{"categories":["Secure","Software"],"contents":"I applaud many of our customers for being smart. Not to say other people are not, but they have made a specific choice in the past based on an understanding. They understand that a single security solution to make your IT environment safe, simply does not exist. It is the combination of tools, or your toolkit, which does. For this same reason, a carpenter has a tool chest, not a single tool.\nAs a founder, I get to see the feature requests. Many of them, which sound great on paper, simply do not belong in our product. Why? We focus on auditing of Unix based environments. So extensive logging features are not part of our product (for that you might want to use Splunk or other tools).\nThese feature requests made me think about the following question: why do we want to have just one single solution for things?\nPros and Cons Some benefits of one solution are immediately clear: good integration, usually cheaper, and less overhead. On the other hand, one solution is often also a compromise on specialization benefits. Another issue with having too much functionality into 1 tool is that it becomes harder to use. After all, more functions have to be implemented, making the user interface harder to use. Going back to the carpenter, he would have to handle a tool so big in size, it is impossible to use.\nMaking Security Simple If you want to make security simple, you should start at the beginning. It is the place where you look at your threats to your business and operations. Second are the involved risks, from business to technical risks. If the threats and risks are clear, you can start with creating your toolkit. You select the right tools for your personal toolkit. Some companies might put additional focus on logging and event management, while others focus on malware.\nThe Unix Way In the field of Unix administration, we apply the rule \u0026ldquo;do one thing, and do it really well\u0026rdquo;. It is for this particular reason why Unix based systems are stable. Each tool is doing one single thing. For unclear reasons, we don\u0026rsquo;t want to apply the same when it comes to security. Maybe because it is still seen as a necessary burden? In any case, there is a lesson to learn from this. Small and simple things, usually are a lot stronger. If you want to have a powerful tool to solve a problem, select the product which is specialized in that.\nBuilding Toolkits If you are building your toolkit, you might wonder where to start. After all, there are so many tools available, both commercially and open source. As an extension to the carpenter analogy, let\u0026rsquo;s go from there. If the carpenter wants to keep his toolkit up-to-date, he will determine what kind of work he did lately and what is there to come. Within the world of security we should do the same. Too often, we rush into making a product purchase while we don\u0026rsquo;t really know what we need.Better planning helps to create\nBetter planning helps to create budget and become more proactive to deal with known and unknown threats. For example, if you are a hosting company, you might not have to deal with malware currently. If you did your risk assessment properly, you will know there is a fairly high risk of websites being infected with spam scripts. So this is a great start for filling up your toolkit with tools.\nJust filling your toolkit with similar products, is a recipe for disaster. Your toolkit should have a variety set of hammers, screwdrivers, and measuring tape. We need tools to measure, like one tool for intrusion detection. Another tool might be there to limit access, or prevent something from happening at all.\nConclusion There is no \u0026ldquo;one size fits all\u0026rdquo; tool when it comes to security. Consider yourself the carpenter who needs to work on different projects, and select the appropriate toolkit for the job. If you are in the process of selecting a new solution, drop the \u0026ldquo;it needs to have all\u0026rdquo; and consider combining more tools. Create your own toolkit, to do your job easier, using the power of each single tool.\nHappy hardening!\n","permalink":"https://linux-audit.com/simplifying-security-choose-the-right-toolkit-not-tool/","tags":["security"],"title":"Simplifying Security: Choose the Right Toolkit, not Tool."},{"categories":["Automation","Containers","Docker"],"contents":"One of the pioneers in the world DevOps, is the company Docker Inc. Known for its toolkit around Linux container technology, they propel the way this technology evolves and is promoted to the world. With great achievements and interest from the outside world, also comes a lot of pressure. Competing products are showing up, resulting in a battle for features, pricing and customers. Unfortunately for security professionals like us, the many security lessons from the past seems to be forgotten. We might be battling the same issues as before…\nDevOps movement In the last few years, the DevOps movement gained a lot of momentum. One of the reasons might be the need for companies to be more “agile”. This includes releasing software quicker and more often. All with the goal of providing higher quality and lower costs at the same time.\nWhile the benefits of DevOps are great, the role of “being a DevOps” might be confusing for the people itself. Those who previously were sysadmins or developers, suddenly find themselves doing work from both worlds. Let’s be honest, it is close to impossible to be an expert in multiple areas, or keeping up with all new developments.\nDo we have a problem? Especially for auditors and security professionals it is hard to keep up with these new technologies. We simply do not have enough hours per week to extensively dive into each new technology. When technology is then also limited to one platform, you have to simply make choices and specialize in one area.\nEven developers and admins who already used Docker, might be confused by all available parameters. Worst, they only seem to increase every new Docker release. It is great to see SELinux support, but didn’t we all turn that off on our host system as well? With the existing time pressure in our work, new features are usually skipped. This is especially true if they take a lot of time to test, deploy and monitor. We all know that usually security features are not in the category “simple and easy” to deploy, without extensive testing.\nDocker and security In the last few releases of Docker, the company showed that security is a subject you cannot simply skip. Some vulnerabilities were patched, and several new security features were introduced. Examples include allowing a limited set of capabilities and the usage of MAC frameworks. By looking at these new options, we can get a glimpse of what is already possible, and where the technology is still immature. Being a DevOps gets easier due to container technology, and at the same time more complicated as well.\nContainers do not contain The “containers do not contain” was a commonly heard phrase. That problem was caused because of a missing namespace for users and groups. For example gaining “root access” within the container, means you got similar privileges on the host system itself. From there it was a small step to compromise the security of the whole machine. With the user namespace this problem is mostly solved.\nAnother example why containers are not fully isolated, is for example keyrings, storing crypto keys. This tooling can’t see the difference yet between UID 80 in one container from another user with the same ID. Due to these constraints, we should still treat containers similar to a normal host system. For example running services under the context of the root user was always considered bad practice. Which it still is, also when using containers.\nNamespaces Namespaces separate several internals of the Linux kernel, which allows it to create different “views” of what a system looks like. This way multiple environments can run on a single kernel, each with its own processes, users, network routing and mounts. It is like a virtual machine, except that containers are simply a single process. This reduces a lot of overhead and provides flexibility when packaging up software. Together with control groups, cgroups for short, the kernel can control processes. With cgroups the priority and resources can be controlled for example. Namespaces separate one big area into smaller ones, cgroups ensure that all areas behave.\nNamespace complexity Docker is actually waiting for the user namespaces to be finished, so it can leverage all its functions and get one step closer to full containment. The first few developments regarding user namespaces are finished and available. For example, the usage of subordinate users and groups is already possible. This functions helps the host system to map users (and groups) within each container, to different users on the host itself. For example user ID 1000 within the container, might be user ID 101000 on the host system. The functionality is definitely much more complex that it looks at first sight.\nOne restriction was the common 16 bits limit for user IDs, limiting it to only 65535. Maybe this restriction is even the easiest part to solve. A little bit more time goes into the adjusting of common userland and helper tools, to deal with the mapping of users. Examples include tools to create, modify or delete users (useradd, usermod, userdel), helper tools (newuidmap, newgidmap) and the usage of new configuration files like /etc/subuid and /etc/subgid. What looks like an easy extension in one file, turns out to affect a lot more files in the end.\nBuild, Ship and Run? Most things in IT start in the building phase. In the case of Docker, you might want to consider a little bit more time in the phase before: preparation. Before just building things, you will benefit from a clear strategy. This starts with how you want to divide applications, and what makes a container actually a container. Right now the consensus seems to be a unit, which has one primary function (e.g. be a database server, or provide a web application). Whatever you choose, ensure that there is a definition in place within your organization. From there start building containers according to that strategy.\nBuilding The building process is one of the most interesting parts. Here images gets build, which then will be used for running new containers. At this stage security awareness and implementation is all depending on the skillset of the builder. Unfortunately, developers usually have a lower urgency to do things the secure way, than most system administrators do. Where the developer has the focus “get it running”, the system administrator cares more about system stability.\nThe Dockerfile Docker build files, usually with the name Dockerfile, are small scripts to guide the build process. They instruct the docker binary how to create an image, and what commands to execute. The first thing is defining the base image, from which the container will be build. Usually defining the maintainer is next, followed up by installing packages. If you will create, tune or analyze a Dockerfile, it is important to know these basic commands, to determine what the container is actually doing. While the commands might have very self-explanatory names, they have small subtleties in them. Just copy-paste an existing Dockerfile and adjust it, will not always give the results you are seeking.\nCommand Function ADD Copy archives, downloads or data into the image CMD Define default command to run (usually the service) COPY Copy data into the image ENV Define an environment variable EXPOSE Makes a port available for incoming traffic to the container FROM Define the base image, which contains a minimal operating system MAINTAINER Maintainer of the image RUN Execute a command or script VOLUME Make directory available (e.g. for access, backup) WORKDIR Change the current work directory Best practices Docker provides extensive documentation regarding the build process, including a best practices document [1]. After analyzing hundreds of build files (Dockerfile), we can conclude that many builders definitely do not follow these best practices. Issues are varying from skipping simple optimization steps when installing software components, up to using “chmod 777” on data directories. If you are using Docker within your organization, analyzing build files will definitely give an idea about the best practices applied within this area. Since we are talking about DevOps and automation, the open source auditing tool Lynis[2] helps you to check for some of the best practices in your Dockerfile.\nSteering the ship Even with lacking security awareness, or missing security features, not all hope is lost. Docker provides a few features:\nSELinux/AppApparmor support - limit processes what resources they can access Capabilities support - limit the maximum level a functions (or “roles”) a process can achieve within the container Seccomp support - allow/disallow what system calls can be used by processes docker exec - no more SSH in containers for just management Additionally we can use iptables, to limit the network traffic streams even further. On the host system, you might apply technologies like GRSEC and PaX, followed by other generic system hardening practices.\nConclusion When we look at the world of vessels and containers, it becomes clear that container technology is not very mature. When we look specifically at the security level, there is even more room for improvement. At least Docker gave both the technology and security awareness a boost, resulting in the first signs of a healthy ecosystem. The existing security features definitely look promising and worth investigating. Let’s hope this article is outdated in a few years. For now, wishing you a great and safe trip.\nThis article was originally published in issue 45 of (IN)SECURE Magazine and has been reposted with permission. After the years, this articles has been updated on this blog.\n","permalink":"https://linux-audit.com/devops-vs-security-can-docker-make-a-difference/","tags":["containers","devops","docker","linux","namespaces","security"],"title":"DevOps vs Security: Can Docker make a difference?"},{"categories":["Accounting","Auditing","Troubleshooting"],"contents":"The Linux Audit framework is a powerful tool to audit system events. From running executables up to system calls, everything can be logged. However, all this audit logging comes at the price of decreased system performance. Let\u0026rsquo;s have a look at how we can optimize our audit rules.\nPerformance tips Good auditd performance will reduce stress on the Linux kernel and lower its impact. Before changing anything to your system, we suggest benchmarking your system performance before and after. This way you can see the benefits of your tuning efforts.\nStrategy: Rule Ordering Placing rules in the right order\nMany software packages use \u0026ldquo;order based rule processing\u0026rdquo;. This means each rule is evaluated, until one matches. For the Linux audit daemon, this principle applies as well.\nSo one of the biggest areas to tune is the order of the rules. Events which occur the most should be at the top, the \u0026ldquo;exceptions\u0026rdquo; at the bottom.\nIf your Linux audit set-up is done alphabetically, you can be assured this configuration is not optimized for performance. Let\u0026rsquo;s continue tuning auditd in some other areas.\nStrategy: Excluding Events Determining what message types are used a lot\nThe challenge with logging events, is to ensure that you log all important events, while avoiding logging the unneeded ones.\nSome administrators apply the \u0026ldquo;just log everything\u0026rdquo; rule. While it often makes sense, it is definitely not efficient and decreases the performance of the Linux kernel. This kind of logging will definitely decrease the processing time of auditd and have a negative impact the performance of the kernel.\nTo enhance the logging, we first need to determine what events often show up.\nMost events sorted by executable\naureport -ts today -i -x --summary\nMost events sorted by system call (syscall)\naureport -ts today -i -s --summary\nThis will reveal what executable or system call is flooding your audit logs. By defining \u0026ldquo;-ts today\u0026rdquo; we only see the recent events.\nThe output of aureport definitely helps to reduce the amount of logging, by disabling some events. Of course you can do this also for events, files and other types. See the man page of aureport for more details.\nSummary of aureport showing events which occurred today\nIgnoring events Now we know what type of files, events or other messages we have, we can ignore them. For that we have to make a rule, which matches and states the exclude of exit statement.\nThe exit statement is used together with syscalls, for others we use exclude.\nFilter by message type\nFor example disabling all \u0026ldquo;CWD\u0026rdquo; (current working directory), we can use a rule like this:\n-a exclude,always -F msgtype=CWD\nAs the first match wins, exclusions have to be placed at the top of the rule chain. As this is a filter based on a message type, we use exclude.\nFilter by multiple rules\nAnother example is suppressing the messages logged by VMware tools. For that we combine multiple rules together, by providing multiple -F parameters. You are allowed up to 64 fields, but usually a few are enough. When using -F, each expression will be checked with a logical AND statement. That means all fields have to be true, to trigger the action of the audit rule set.\n-a exit,never -F arch=b32 -S fork -F success=0 -F path=/usr/lib/vmware-tools -F subj_type=initrc_t -F exit=-2\n-a exit,never -F arch=b64 -S fork -F success=0 -F path=/usr/lib/vmware-tools -F subj_type=initrc_t -F exit=-2\nNote: some examples might have different results on older machines. Therefore always test each rule to determine if it works. Rules which don\u0026rsquo;t do anything are only negative for performance.\nStrategy: Determine Buffering Needs Tuning buffer needs for auditd\nBy default the auditctl can provide some statistics when using the -s (status) flag. It shows its status (enabled), any related flags, process ID and log related statistics (backlog, rate, lost).\n# auditctl -s enabled 1 flag 1 pid 430 rate_limit 0 backlog_limit 320 lost 0 backlog 0 enabled -1 flag 0 pid 0 rate_limit 0 backlog_limit 320 lost 0 backlog 0 Allowing bigger buffers means a higher demand on memory resources. Depending on your machine this might be a small sacrifice, to ensure that all events are logged.\nTo determine the best possible buffer size, monitor the backlog values. They should not exceed the backlog_limit option (in our case 320). Another useful statistic is the monitor the lost value, as it will tell you how many events could not be processed. On a normal system, this value should equal or close to zero.\nStrategy: Monitoring Directories Use path instead of dir when monitoring a specific directory\nThere are two ways to monitor the contents of a directory: path or dir.\nDepending on what you want to monitor, monitoring subdirectories might not be needed. In such case it is better to use the path option, as it monitors only that directory. It\u0026rsquo;s a small adjustment, which might save you a lot of unneeded audit logging.\n","permalink":"https://linux-audit.com/linux-audit-framework/tuning-auditd-high-performance-linux-auditing/","tags":["accounting","auditctl","auditd","auditd.conf","performance","system performance","system tuning"],"title":"Tuning auditd: high-performance Linux Auditing"},{"categories":["Vulnerabilities","Web"],"contents":"Important Note This is an older blog post and we no longer advise using Web of Trust. See pcmag for more details.\nProtecting the web browser Usually we focus on the blog on the server side of things, helping to protect the data of users, customers and ourselves. What we commonly overlook is the end of the connection, the web browser of the user. In the upcoming posts we will look at alternative measures we can take, to protect data there as well.\nMalware, spam, scam? As we all know, the web is full of good things. But unfortunately it happens also to be a breeding ground for harmful software, scams and unwanted messages. To counter this, there is a project called Web of Trust (WOT). It uses the social part of people to share insights whenever a site is good or bad. In other words, it leverages trust people express in a website.\nHow it works When visiting a website, the WOT plugin will check the status of a website. If enough people marked it as being a bad website, the plugin will interrupt you and ask what you want to do. This way malicious websites are intercepted quickly, before being able to do any harm to your browser.\nAs you can see in the example above, search results can be labeled as well. This way you know to avoid a website, before even clicking the link.\nLinux support Web of Trust is available as a plugin for most browsers, including Firefox and Chrome for Linux. This makes it an interesting addition to your own browser if you use Linux. However, since it affects web browsers, it can also be used on the systems of your family members.\nInteresting is the how data is also shared with privacy aware search engines like DuckDuckGo. If you care about your privacy and want to remain as safe as possible on the web, this is definitely a powerful combination.\nDownload WOT Web of Trust can be downloaded freely from their website, or via the add-ons of your browser.\nImportant note: see the note on top.\n","permalink":"https://linux-audit.com/protecting-the-browser-web-of-trust/","tags":["malware","one-time"],"title":"Protecting the browser: Web of Trust"},{"categories":["Hardening"],"contents":"Most of the security defenses on Linux, are based on the earlier performed hardening activities. By locking down components on the system, the chance of a full compromise is lowered. This step-by-step locking down is a time consuming process. Time to review some of the strategies which can be applied when you want to secure your systems.\nStrategy 1: Locking down processes The first area to lock down are system processes. After all, each system needs processes, to fulfill its roles. It is common to see systems are running too much processes. Not in the sense of numbers, but mainly the required processes including non-functional processes. These last group of processes have a clear goal, but are not applicable for the machine. It might be a NFS daemon which is simply not used. Another example might be a power saving utility that runs on a virtual system, while it may only work on the host system.\nWhen to use Processes can be locked down in different way. From memory management, disk and file permissions, up to the resources. This strategy is in particular useful for systems which allow its user to start their own processes. Another type of system which benefit is systems being part of a web cluster, or act as individual web servers.\nExecution: processes To execute this strategy, follow at least the next steps:\nDetermine which processes run as root Check which users can start processes on the system, directly or indirectly Determine an appropriate security framework like SELinux or AppArmor Determine allowed directories/files, limit access as much as possible Limit file descriptors, memory and other system resources which can be quickly exhausted Locking down networking For most information nowadays, it enters and leaves the system via the network. Therefore this is another area which needs locking down. The most obvious measure is traffic filtering, which can be achieved with iptables tooling. For newer systems, this might be replaced with nftables. Both are possible with the kernel feature named Netfilter.\nWhen to use This strategy is useful for most systems, especially those directly or indirectly being available via the internet. The strong area in this strategy is the focus on allowed paths, while (by default) blocking all others. Traffic you don\u0026rsquo;t want, should not be able to enter or leave the system.\nExecution: networking Configure a firewall Disable interface not used or allowed (e.g. WiFi) Limit ARP, DNS and routing to allowed paths only Configure logging for possible intrusion detection or forensics While an initial set-up might take a while to analyze, configure and tune, it can be reapplied on other systems with similar functionality. Since this strategy is useful for almost all systems, it is well invested time.\nLocking down users Users and processes are often considered \u0026ldquo;subjects\u0026rdquo; on a system. They are active entities, who want to use \u0026ldquo;objects\u0026rdquo;, like a file. While not every system has users directly on the system, it often has indirect users. This may be an anonymous user visiting the website, or an authorized user which submits a print job.\nWhen to use This strategy is especially important for systems with local user accounts. Without proper measures, they might be able to start programs, or access data, they are not supposed to. For other systems basic measures should be taken.\nExecution: users Determine allowed users Set default permissions Limit access on directories and files with sensitive data Properly implement authentication, authorization and accounting measures. Usage of PAM, Linux audit framework and logging. Locking down the kernel The inner part of the system, the kernel, is responsible for resource management. It will check the available resources and determine what a subject can do to an object. In other words, is a user allowed to access file X, or can a network packet be accepted for a later delivery at a service.\nWhen to use This strategy is useful for almost all systems. It does however require a more defined subset of measures. For example the usage of the earlier mentioned SELinux or AppArmor. Not all systems require this, yet it can be a very powerful mechanism to restrict access to authorized requests.\nExecution: kernel Determine if SELinux or AppArmor can be used Check what kernel settings should be altered, to prevent resource exhaustion Determine the importance of file integrity and what measures should be taken Examine the need for auditing resources with Linux audit framework Locking down software Every system has software installed. Unfortunately systems are often not properly patched, or installed with the default settings. The result is a system with vulnerable software and another possible hole in which data can be leaked to unauthorized people.\nWhen to use This strategy in particular should always be properly investigated. The only exception might be fully isolated systems, with a dedicated and standalone goal. Even then, the risks should be well-known and proper measures taken. For the other 99.9% of systems, a proper patch management process and configuration hardening should be performed.\nExecution: software Regular software patching Checking for vulnerabilities Perform audits on configurations to determine weaknesses Conclusion Every system is different. When you want to lock down a Linux system, ensure that the primary role of the system is known. Then select the right measures to go along this role. In the end security is about maximizing availability, integrity and confidentiality. While some performance hits might be expected, the system should always be focused on fulfilling its role.\n","permalink":"https://linux-audit.com/lock-down-strategies-for-linux-servers/","tags":["apparmor","hardening","selinux","system hardening"],"title":"Lock Down Strategies for Linux Servers"},{"categories":["Auditing","Automation","Compliance","Hardening"],"contents":"The Center for Internet Security, CIS for short, is the organization behind several in-depth hardening guides. The quality of these hardening guides is outstanding, with a high level of detail.\nThis high level of detail has one downside: it costs a lot of time to read, try and test the recommendations. Sometimes we simply don\u0026rsquo;t have the time to do an extensive audit by hand. Let alone the time to actually repeat the auditing and hardening steps on a regular basis. Fortunately there is a solution: tooling.\nCIS has their own CIS-CAT auditing tool. Unfortunately this is out of reach for most of us, as membership is expensive. For companies and individuals who seek to do auditing, yet want an alternative to the paid CIS-CAT tool, might be delighted to know there are several open source options available.\nVulnerability management Depending on your organization, size and type of business, there are a lot of tools available to assist you. For example the well known port scanner Nmap. During the years it became much more than just port scanning. For example with the use of plugins, you can use it to test for new vulnerabilities.\nWhen we continue within the area of vulnerability management, we can\u0026rsquo;t ignore the great project OpenVAS. As an original fork of the now commercial Nessus, it helps with finding vulnerabilities on your system. While it may be different than using a hardening guide, it will definitely discover other issues and more quickly.\nLinux system auditing When it comes to a closer alternative of CIS-CAT tooling, we can\u0026rsquo;t ignore our own tool Lynis . Like Nmap and OpenVAS, it is open source and freely available. It helps automating the system auditing process. On top of that, it provides feedback for further hardening of your Linux systems.\nFor those who search compliance checking (e.g. PCI DSS), we are sure that Lynis will be a great help. If your goal is \u0026ldquo;easy hardening\u0026rdquo;, we suggest to start creating your customized scripts. Don\u0026rsquo;t fall into the trap of a false sense of security. We covered this in our post about the possible backfire of hardening scripts.\nMore alternatives GitHub Another great research nowadays are the snippets found on GitHub. It must be said that some repositories are simply a mess, but when searching you might find some gold nuggets. Most of these snippets are provided by passionate people, who like to share their knowledge. The caveat is that some snippets are not up-to-date, wrong or might not work. But depending on what you try to achieve, that might be simple to check.\nYour OS distribution Most Linux distributions have their own hardening guides available. While not a direct replacement for tooling, it might be a combination. For example mixing OpenVAS, Lynis, Nmap and the guides from both CIS and the distribution.\nDepending on your OS, the quality might be different. So even if you are not using Fedora, it might be still worth checking out their resources. Also Arch and Gentoo are known for sharing a lot about security related topics.\nLinks Order by alphabetical order, some useful links:\nLynis Nmap OpenVAS CIS Red Hat hardening guide Ubuntu security tips Do you know any other open source tools, as a simple alternative to CIS-CAT? Love to hear!\n","permalink":"https://linux-audit.com/find-the-alternatives-cis-cat-auditing-tool/","tags":["audit","fedora","gentoo","hardening","lynis","nmap","openvas"],"title":"Find the alternatives: CIS-CAT auditing tool"},{"categories":["Auditing","Lynis"],"contents":"When auditing a server, it may be useful to only run a particular category of tests, like firewall related tests. In that case the -tests-category parameter can be used, together with the category name.\nAvailable categories To determine what categories are available, Lynis has a built-in parameter -view-categories which lists all available files. Most of the names are self-explanatory on what of tests they include. For more information about the included tests, have a look in the ./include directory, where files are listed as tests_.\nExample\n# lynis --view-categories [+] Available test categories ------------------------------------ - accounting - authentication - banners - boot_services - crypto - databases - file_integrity - file_permissions - filesystems - firewalls - hardening - hardening_tools - homedirs - insecure_services - kernel - kernel_hardening - ldap - logging - mac_frameworks - mail_messaging - malware - memory_processes - nameservices - networking - php - ports_packages - printers_spools - scheduling - shells - snmp - solaris - squid - ssh - storage - storage_nfs - tcpwrappers - time - tooling - virtualization - webservers After selecting which category you want to use, simply run Lynis again:\nlynis audit system --tests-category firewalls\nThis will tell Lynis to run all firewall related tests and skip the other categories.\n","permalink":"https://linux-audit.com/lynis/viewing-available-test-categories-in-lynis/","tags":["lynis","tips"],"title":"Viewing available test categories in Lynis"},{"categories":["Automation","Containers","Docker"],"contents":"Docker simplifies software packaging by creating small software units. It starts with a base OS image, followed by software installation and finally the configuration adjustments. For building your own images, Docker uses small build files, with the less than original name Dockerfile.\nDocker build files simplify the build process and help creating consistent containers, over and over. Unfortunately developers don\u0026rsquo;t always take security into account during the build process, resulting in software which is installed insecurely. In this post we have a look at how to improve several areas within the build process and secure software properly.\nBasics Normally Docker build files are named Dockerfile and contain a set of instructions to build an image. With newer versions of Docker you can alter this name, but for convenience the default name can still be used. The building itself is done with the docker build command, which parses the build file and instructs what steps should be performed to build your custom image.\nDocumentation While a Dockerfile may look simple for the author of the file, the includes steps aren\u0026rsquo;t always seem logical for others. Therefore it is wise to implement the following components:\nMaintainer Comments Version Control Maintainer Usually a file has an owner, or maintainer of the file. By specifying the name and contact details, other developers or users of the software can make suggestions. While a Dockerfile seems to be perfect right now, it may be less optimal in the future.\nRelated Dockerfile argument: MAINTAINER Firstname Lastname \u0026lt;e-mail address\u0026gt; \u0026lt;other information\u0026gt;\nComments Like good source code, scripts and build files should be properly documented as well. With the # sign, lines can be ignored by the build process, while at the same time give valuable information about the steps involved to readers of the file.\nVersion Control Most developers already use tools like Git to maintain software versions. With the Dockerfile being an important part of the build process, this file definitely needs a place as well.\nInstallation of software Usually one of the first steps in a Docker build file, is the installation of the required software components. The best practices describe to run first a repository update, followed by a chained installation process. What this means is combining several commands, to properly use the caching mechanism and at the same time stop if 1 of the commands in the chain fails.\nWrong method RUN apt-get update RUN apt-get -q -y install lynis This is wrong because it may result in caching issues, which will effect proper execution of the second command.\nGood method RUN apt-get update \u0026amp;\u0026amp; apt-get -q -y install lynis When using just installing a few packages, you might want to put everything on one line. However, when several packages needs to be installed, terminate the line with a backslash and start at the next line.\nRUN apt-get update \\ apt-get -q -y install lsof \\ lynis If you want to do things properly, sort lines for easier reading and clean up after you are done installing the packages. This can be done by adding cleanup steps to the chain: \u0026amp;\u0026amp; apt-get clean \u0026amp;\u0026amp; rm -rf /var/lib/apt/lists/*\nRUN apt-get update \\ apt-get -q -y install lsof \\ lynis Repositories When possible use the original repositories. They are tuned for optimal performance and minimal in size. Sure, you could save a few bits here and there, but it\u0026rsquo;s a minimal gain. Creating your own base image, also means you need to keep it up-to-date.\nOpening network ports Most software components listen to a network port for communications. This may be frontend traffic for the related user traffic (e.g. web server), or backend traffic like a database connection.\n# Expose SSL default port EXPOSE 443 Limit the amount of ports only to what is strictly needed for accessing the services. Try to avoid opening up debugging interfaces, or other backdoors. Fine for development, but make sure it won\u0026rsquo;t end up in your production environment.\nInstallation of external software components Almost every container needs software. For internal created components, they can be copied into the image (e.g. with the ADD or COPY statement).\nCopying files When adding files into the image, the COPY statement is preferred. To ensure proper usage of the cache, separate COPY statements (opposed to package installation activities). This helps in performance, by invalidating just some parts of the cache.\nDownloading files Most software is available online, which means it has to be downloaded. While there is nothing wrong with downloading files, we need to be fairly sure that what we have downloaded, is what we think it is. In other words, we need to ensure the integrity of the file download as a minimum. Even better is if we can check the authenticity of a file, by using signed software packages. Usually bigger software packages provides their downloads via HTTPS and with a signature.\nThe worst possible scenario for a download in a Dockerfile, is it via HTTP only, without any checking. Unfortunately this still occurs on a regular basis, making these builds susceptible to man-in-the-middle attacks.\nUse cURL/wget instead of ADD\nTo limit the size of an image, the usage of cURL or wget is preferred. By using ADD files will be extracted into the image, increasing size. With the goal of keeping things to a minimum, it is better to use the other tools. Additionally, the command can be appended directly with an integrity check, which is not possible when using ADD.\nDisk, Directories and Mounts The Docker build file allows defining storage areas for your application with the help of the VOLUME statement. Only add those directories which are necessary. Keep things as small and limited as possible. Again, document why this path is required.\nWorking directory Instead of using the combination of \u0026ldquo;cd /data \u0026amp;\u0026amp; ./runscript.sh\u0026rdquo;, the WORKDIR statement changes the current work directory. This helps with readability and simplifies auditing Dockerfiles.\nRunning Processes Processes can be started with the CWD statement. For example starting Lynis:\nCMD [\u0026ldquo;lynis\u0026rdquo;, \u0026ldquo;-c\u0026rdquo;, \u0026ldquo;-Q\u0026rdquo;]\nEnsure that your process in the path, or use the full path.\nEnvironment settings By using the ENV statement, we can define environment settings. A common one is to define the path, for your custom binary location.\nENV PATH /usr/local/yourpackage/bin:$PATH\nBe aware that environment variables won\u0026rsquo;t always work the same under different shells or on other platforms.\nActive User When possible, the least amount of permissions should be used, also during execution commands. With the USER statement, the permissions can be dropped from root to a non-privileged user.\nAuditing tool for Docker After reading these tips, you might want to check your files. Wouldn\u0026rsquo;t it be great if there was a tool to do this for you? Well, gladly there is a \u0026ldquo;Docker auditing tool\u0026rdquo;. Download the free Lynis tool and audit it with:\nlynis audit dockerfile \u0026lt;file\u0026gt;\nThis command will initialize the Docker related tests and performs a security related scan on the specified Dockerfile.\nConclusion If you want to create solid and secure Docker build files, these are the things you should do with your Dockerfile:\nAdd a maintainer Combine different apt/yum commands by \u0026ldquo;chaining\u0026rdquo; them. Document the file properly and use versioning. When possible download files via HTTPS, use signed software packages or have at least a checksum validation. Set your permissions of files as tight as possible. No chmod 777, keep that for your development system. Got more tips for safe Dockerfiles? We love to hear!\n","permalink":"https://linux-audit.com/security-best-practices-for-building-docker-images/","tags":["docker","security"],"title":"Security Best Practices for Building Docker Images"},{"categories":["Auditing","Automation","Compliance"],"contents":"Increased strength when combining tools for automation and security of IT environments\nTools like Ansible, Chef, and Puppet are used a lot for rapid deployment and keeping systems properly configured. These tools in itself are great for ensuring consistency over your systems.\nSo what is Configuration Management? Configuration management is the art of keeping systems properly configured. Usually companies start small, which equals manual configuration. Each time a new system is deployed, it is configured manually. While there is nothing wrong with this, it becomes an issue when systems are not kept up-to-date.\nThe earlier mentioned tools help with orchestrating how systems should be configured. This ranges from installed packages, up to specific configuration settings. Even software patching can be performed, simplifying the process of keeping systems up-to-date.\nExample output of Ansible automation tool\nConfiguration management: When to Use? The best moment to start using configuration management tools, is when you realized systems are different (while they were not supposed to be). Usually starting from 20-25 systems and upwards, automation of configurations will beneficial in the long term.\nSpeed\nAnother clear benefit is the speed of deployment. After all, almost no manual steps are needed. So environments which rely on this speed, are definitely a good candidate to use configuration automation.\nDiversity\nCompanies with a lot of diversity in their operating systems, might have less benefit from configuration management tools. After all, a lot of exceptional configurations have to be made, specifying different ways to get the same result. Even the smallest action like installing a package, is a totally different set of commands between each operating system.\nPicking the right tool(s) When it comes to the differences in automation tools, there are several important areas. These are mainly the underlying programming language, the structure of files, and the way the communication occurs between the central server and the agents.\nPersonal preference\nUsually a lot comes to the preference of the system administrator, which has to use the tool. If he/she has a strong preference for Python, a tool like Ansible might be more attractive. This simplifies installation (e.g. using pip), but also when implementing more advanced scripts. Sometimes the logic of the underlying programming language is visible there.\nWhen selecting a tool, we suggest to have a look at the following attributes:\nPricing Community support Availability of snippets Simplicity of tooling, website and documentation Preference of programming language Security Automation Configuration management tools are also great on supporting security objectives. One area might be system hardening, in which the tooling ensures that some settings are always enforced. Even if a system administrator or developer changes a setting, it will be reversed into the preferred setting.\nContinuous Auditing By combining configuration management and auditing, we can close the loop of automation. It enables us to perform configuration management, continuous auditing, and security monitoring at the same time. Most of the gaps will be closed by one tool, while the other one keeps an eye on existing and new risks. If for some reason something can be tuned, it will show up on the auditing side. It will be then an easy step to feed this as input and auto correct the issue.\nDeploying an auditing tool It should be a surprise that also an auditing tool could be deployed automatically. In the screenshot we can see how our auditing tool Lynis is installed. After installation it will be also configured and scheduled for execution. In our case we close the loop even further, by uploading the data to a central node, and monitor for regular audits. If the central system does not receive data for a few days, something is wrong and need attention. In other words, it equals a failure somewhere in a the chain. Only then we need to do a manual check. This type of automation prevents \u0026ldquo;ghost\u0026rdquo; systems, and solves malfunctioning systems or software, which otherwise would get noticed after months\u0026hellip;\nIn the upcoming time we will definitely blog some more about automation and auditing.\n","permalink":"https://linux-audit.com/security-integration-configuration-management-and-auditing/","tags":["automation","hardening","linux"],"title":"Security Integration: Configuration Management and Auditing"},{"categories":["SSH","System Administration"],"contents":"Linux systems are usually managed remotely with SSH (secure shell). Still many administrators are using passwords, instead of keys. Keys not only boost security, it also makes managing systems much easier. Instead of entering your password for each server, you only have to do it once per session. When managing several systems per day, you will be wondering why you ever used password based authentication before.\nGenerating the SSH key Depending on your desktop platform, we first have to create a key pair. This will consist of a public and private key, which are both needed to work. The private key is private and should remain private. While not mandatory, it is very wise to protect it with a password. The public key will be configured on the remote system. This key is not secret at all, therefore it can safely stored on another machine, or even shared with others. Since people still make the mistake of sharing the private key, reassure yourself what key you are sharing at any given time.\nPuTTY Windows users can use the PuTTYgen utility to create a key pair. This tool is part of the full installation of PuTTY, or can be Downloaded manually.\nPuTTYgen creating the keypair\nssh-keygen For users who will do management from a central system, or run Linux (or any other Unix based system), can use ssh-keygen. If you need to support recent OS versions, it is suggested to use the newer Ed25519 key format. Otherwise, use RSA.\nssh-keygen -o -t rsa -b 4096 -C \u0026quot;myname@example.com\u0026quot;\nThe output would look something like this:\nssh-keygen command creating a 4096-bit RSA key\nNow check your created key and see if it is of the right type and bit size.\n# ssh-keygen -l -f .ssh/id_rsa 4096 98:eb:9a:f7:94:bf:a0:a1:4b:55:ca:82:c3:24:46:b8 .ssh/id_rsa.pub (RSA) As you can see in this example, the tool will select the public key, even if you don\u0026rsquo;t provide they private key.\nCopying the key Next step is copying the key to the other system. The easiest way is using the ssh-copy-id command. Just provide it with the \u0026ldquo;ID\u0026rdquo; to copy and your username and hostname of the remote system.\n# ssh-copy-id -i ~/.ssh/id_rsa.pub michael@192.168.1.251 The authenticity of host \u0026#39;192.168.1.251 (192.168.1.251)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is b7:39:02:6a:f3:be:42:c3:d8:69:c4:7f:4e:9b:0b:f3. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys michael@192.168.1.251\u0026#39;s password: Number of key(s) added: 1 Now try logging into the machine, with: \u0026#34;ssh \u0026#39;michael@192.168.1.251\u0026#39;\u0026#34; and check to make sure that only the key(s) you wanted were added. [root@archtest .ssh]# ssh \u0026#39;michael@192.168.1.251\u0026#39; Enter passphrase for key \u0026#39;/root/.ssh/id_rsa\u0026#39;: Last login: Sun Dec 21 23:49:57 2014 from arch\u0026lt; Another option is to do it manually, or copy it via SCP. These steps can also be used when you created a key for PuTTY. If you used PuTTYgen to create the key, it will give you the string to add to the authorized_keys file. Something like \u0026ldquo;ssh-rsa AAAA[long string]= rsa-key-20150316\u0026rdquo;.\n# ssh username@remote-system mkdir ~/.ssh chmod 700 ~/.ssh edit ~/.ssh/authorized_keys and copy the public key chmod 600 ~/.ssh/authorized_keys\u0026lt;/pre\u0026gt; Now try logging in and see if your key based authentication is working.\nUsing an agent By using an agent utility, we can leverage caching of our credentials. The ssh command (or PuTTY) does not have to ask us each time the passphrase, but requests it from the agent.\nPuTTY agent (pageant) When using PuTTY, the nifty utility pageant is the PuTTY authentication agent. Start the utility and right click on the icon in the task bar to add a key. Provide your password and that\u0026rsquo;s all.\nWhen logging in with the agent, we see something like \u0026ldquo;Authenticating with public key \u0026ldquo;rsa-key-20150316\u0026rdquo; from agent\u0026rdquo;.\nIf you can\u0026rsquo;t log in without password:\nThe key was not accepted (see event log within PuTTY) The authorized_keys file has incorrect file permissions PuTTY is not configured to use the SSH agent With ssh-agent First run the ssh-agent.\n# ssh-agent SSH_AUTH_SOCK=/tmp/ssh-zo47izH0ZcYM/agent.1133; export SSH_AUTH_SOCK; SSH_AGENT_PID=1134; export SSH_AGENT_PID; echo Agent pid 1134; This output can be used to configure the SSH agent. For most systems the following steps can be used to use the agent.\nFirst use eval to determine if the agent is running:\neval $(ssh-agent)\nNext is to add this to your .bash_profile (or your other shell configuration files).\necho 'eval $(ssh-agent)' \u0026gt;\u0026gt; ~/.bash_profile\nNow we add the key to the agent cache with ssh-add.\nssh-add ~/.ssh/id_rsa\nIt should respond with a message like \u0026ldquo;identity added\u0026rdquo;. Now you can use ssh and connect to your configured system(s) without a password.\nSystemd When using systemd, you might want to create a systemd service file.\n/etc/systemd/system/ssh-agent.service\n[Unit] Description=SSH key agent [Service] Type=forking Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket ExecStart=/usr/bin/ssh-agent -a $SSH_AUTH_SOCK [Install] WantedBy=MyTarget.target Add the following line to your shell .profile file (e.g. .bash_profile):\nexport SSH_AUTH_SOCK=\u0026quot;$XDG_RUNTIME_DIR/ssh-agent.socket\u0026quot;\nNow enable and start the service:\nsystemctl enable --now ssh-agent.service\nServer configuration If you only want to use keys, you can now disable password based authentication. Ensure that your configuration is working correctly, before doing so. When you are sure, update your /etc/ssh/sshd_config configuration file.\nPasswordAuthentication no ChallengeResponseAuthentication no Last but not least, restart your SSH daemon.\nsystemctl restart sshd.service\nHappy hardening your SSH configuration!\n","permalink":"https://linux-audit.com/ssh/using-ssh-keys-instead-of-passwords/","tags":["linux","login","openssh","ssh","ssh-copy-id","ssh-keygen"],"title":"Using SSH keys instead of passwords"},{"categories":["Performance","System Administration","Web"],"contents":" Recently we changed our corporate website into a \u0026ldquo;HTTPS only\u0026rdquo; version. Most of the content is not secret information, still we have some sensitive areas. The ordering section and downloads, and additional our portal. While some areas were already covered with a lock, we felt it was time to make the jump to cover it all.\nAdditionally, we believe that we doing everything we can on our website, practicing security hardening ourselves. So that includes buying a SSL certificate, configure our web servers and finally tune it. In this article we share what we learned while doing so.\nNginx Configuration For the purpose of demonstration, we will show some snippets in this article. While most of it is focused on nginx, the general rules can be applied to others like Apache. In any case, don\u0026rsquo;t simply copy snippets, but test them carefully and understand what they are doing.\nLet\u0026rsquo;s start hardening and tuning!\nDisable old protocols SSL version 2 and 3 are insecure. Several weaknesses in the last years showed that you should no longer use these. Also for companies who need to be PCI DSS compliant, are enforced to remove support for SSLv3.\n# Only allow TLS ssl_protocols TLSv1.2 TLSv1.3; Select right ciphers Ciphers are part of the full conversation, deciding how the connection is initiated and maintained, but also how data is encrypted and protected.\nSelecting the right ciphers is not easy. All kind of vulnerabilities showed that selecting a wrong cipher can weaken your security defenses. Instead, use the great page of Mozilla, which helps you selecting the right cipher set , tuned to modern browsers.\n# Use specific ciphers and let client decide ssl_ciphers \u0026#39;long string of ciphers\u0026#39;; ssl_prefer_server_ciphers off; New HTTPS features Last years several new concepts have been introduced, to make the web a safer place.\nForward secrecy The easy explanation for this feature: ensure that in the event of a private key is exposed previous recorded messages can not be decrypted. The more technical background is that no additional keys can be gathered, when another key is compromised.\nOCSP stapling Instead of just relying on clients checking for a revocation list, we can help saving a lot of bandwidth with OCSP stapling. The server will do the checking instead and stamp it with a recent check. This way the client knows the certificate is still valid.\nHTST HTST, or HTTP Strict Transport Security, helps the browser to save a preferred protocol (HTTPS). This means that after a redirect from HTTP to HTTPS, the browser next time remembers to go directly to HTTPS. This helps with ensuring the right protocol is used while limiting unneeded redirects at the same time.\n# Redirect other domains, including www.domain.com server { listen 80; index index.html index.htm; server_name *.domain.com *.other-domain.com; add_header \u0026#34;Cache-Control\u0026#34; \u0026#34;public, max-age=31536000\u0026#34;; return 301 https://domain.com; } See also the article HTST\nPublic Key Pinning\nHTTP Public Key Pinning, or HPKP is a way to glue a hostname and a public key (of the certificate). This is done at the level of the browser, which has static lists.\nPerformance Every second counts on the web. So where possible, allow clients to cache data and do compression.\nCompression First level of performance tuning is sending less data on the line, with the help of file compression. For this the common used gzip module is used, which all modern browsers understand as well.\nSo we enable gzip, define what gets compressed, how much and how much buffering we use.\n# Turn on gzip gzip on; # Also zip proxied requests gzip_proxied any; # Compression level, only do zipping if minimum length is 1100 bytes gzip_comp_level 6; gzip_min_length 1100; gzip_buffers 16 8k; # Define gzip for specific types (we don\u0026#39;t zip fonts) gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript; # Enabled (insert Vary: Accept-Encoding header) gzip_vary on; Caching While compression is great, it is even better if we limit the amount of requests to our server. We want after all security AND performance.\nWhile browsers are already smart in knowing which files have been changed, it still sends out requests.\nThe orange triangles show that requests have been done and returned with a 304 (not modified). The green circle is a successful 200 response (Ok) of the main test page.\nSo we want to reduce the amount of requests to a bare minimum. So we picked an expire date of 1 week. Long enough for normal users which return on a regular basis, while not caching entries too long. After all, when releasing new content, we want to push that to the client.\nlocation ~* \\.(?:css|js) { root /data/website/; expires 1w; add_header Cache-Control public; add_header Vary Accept-Encoding; access_log off; } This configuration sets both an encoding (for compression) and sets caching to 1 week. That means that only once a week a new copy of the file is retrieved, unless the user forces a new download (CTRL+F5). Since we don\u0026rsquo;t care about logging the request of CSS and Javascript files, we don\u0026rsquo;t log it. Another slight performance win in that area as well.\nHTTPS tuning When it comes to HTTPS, a slow connection is usually caused due to the slow handshake. The main reason is network latency, in combination with the amount of packets needed to finish the handshake. The more packets, the longer it takes. The HTTP/2 protocol can help\nEnable HTTP/2\nEnabling HTTP/2 is easy, just add it to the listen statement. Note, you need a recent version of nginx.\nlisten 443 http2; listen [::]:443 default ipv6only=on http2; Conclusion After all this tuning, we can learn a few things:\nSSL versus TLS\nSSL is old and insecure. Therefore version 2 and 3 should be disabled. TLS is fine, with TLSv1.1 and TLSv1.2 being preferred. TLSv1 might be still needed, for older browsers or some tooling (e.g. some versions of wget).\nUse HTTP2\nMake use of the HTTP/2 protocol to enhance the speed of HTTPS connections.\nSecurity AND Performance\nInstead of security battling against performance, the combination is possible.\nTest your SSL configuration SSL Labs . Use the Gzip module for nginx Use website speed testing tools like Pingdom tools to determine any bottlenecks. Other resources Chrome Internal link for Chrome to check HTST database: chrome://net-internals/#htst\n","permalink":"https://linux-audit.com/optimize-ssl-tls-for-maximum-security-and-speed/","tags":["hpkp","hsts","nginx","performance","security","ssl","system tuning"],"title":"Optimize SSL/TLS for Maximum Security and Speed"},{"categories":["Automation","Software","System Administration"],"contents":"To counter the biggest threat to software packages, they should be updated on a regular basis. Vulnerabilities are discovered on a daily basis, which also requires we monitor daily. Software patching takes time, especially when testing and reboots are needed. Fortunately, systems running Debian and Ubuntu can use unattended-upgrades to achieve automated patch management for security updates.\nInstallation With most software packages, unattended-upgrades has to be installed.\napt install unattended-upgrades\nIf you are not logged in as the root user, use the sudo command to get temporary privileges. Since configuration is needed, we suggest to switch to root and install the package. The only exception is when you directly deploy your configuration with a tool like Ansible, CFEngine, Chef or Puppet.\nUnattended-upgrade and Unattended-upgrades While the package is named unattended-upgrades, the actual script to perform the upgrade is named unattended-upgrade. To avoid confusion, unattended-upgrades (with s) is actually a symlink to the script.\nConfiguration After installing, it is time to configure the package. Although there aren\u0026rsquo;t many things to configure, the configuration file is named /etc/apt/apt.conf.d/50unattended-upgrades. By default, only security upgrades will be installed, which is what most people want.\nNext step is to configure the package:\ndpkg-reconfigure --priority=low unattended-upgrades\nSelect that you want to have stable packages installed.\nInteractive usage First time when testing the package, use the -v parameter. This will the actions on screen.\nUnattended-upgrade in action\nLogging By default all actions are logged to /var/log/unattended-upgrades/unattended-upgrades.log. Information is also available per day, when actual upgrades are found and installed. In this case data is logged to the /var/log/unattended-upgrades directory, with a name similar to unattended-upgrades-dpkg_2015-03-09_18:17:39.573099.log.\nLog rotation Log are rotated via the logrotate service. Usually no action is needed to ensure that the files are rotated as well. It is still advised to check /etc/logrotate.d/unattended-upgrades for more details and confirm things are properly configured.\nRebooting Although software packages are maintained this way, we still need to reboot systems. The package does actually enable the possibility to reboot the system for you:\n// Automatically reboot *WITHOUT CONFIRMATION* // if the file /var/run/reboot-required is found after the upgrade Unattended-Upgrade::Automatic-Reboot \u0026#34;true\u0026#34;; // If automatic reboot is enabled and needed, reboot at the specific // time instead of immediately // Default: \u0026#34;now\u0026#34; Unattended-Upgrade::Automatic-Reboot-Time \u0026#34;02:00\u0026#34;; If you rather do it manually, use the file /var/run/reboot-required, to determine if a reboot is needed.\nOr if you want to know why a reboot is needed, check /var/run/reboot-required.pkgs. Often a reboot is needed due to updates to the Linux kernel, functions (libraries) or other software, which is integrated closely with the kernel.\nSee our related blog post: how to check for a required reboot for Debian, Ubuntu, and others\nTips Although unattended-upgrades helps with simplifying patch management, some remaining actions are left. Some tips while setting it up:\nMonitor for reboot Configure your monitoring tool (e.g. Nagios) to monitor for the presence of /var/run/reboot-required.pkgs. If the file is available, then send out an alert. This way the system administrators know a reboot is needed and downtime can be planned.\nAdditionally, monitor also for uptime. While Unix based systems are stable, it is not wise to let them run for a year. At least patch regularly and when a reboot is needed, to schedule it. It is better to \u0026ldquo;plan for failure\u0026rdquo; and ensure systems can be rebooted easily, knowing other systems take over functionality.\nAudit system on a regular basis Automation can fail. In other words: trust, but verify. Regularly auditing the configuration and proper functioning of the tool, is advised. Tools like Lynis can help with automating this audit process.\nSubscribe to the mailing lists Even if software is simplifying the patch management process for you, the key is in knowing what actual threats endanger your systems. Subscribe to the security mailing lists:\nDebian: Debian security list Ubuntu: Ubuntu security list Happy updating!\n","permalink":"https://linux-audit.com/using-unattended-upgrades-on-debian-and-ubuntu/","tags":["automation","debian","dpkg","patch management","software patching","ubuntu","unattended-upgrades"],"title":"Using unattended-upgrades on Debian and Ubuntu"},{"categories":["Web"],"contents":"OCSP stapling is a logical follow-up on OCSP . OCSP itself just checks if certificate is still valid by determining if it is on a revocation list.\nThe original OCSP protocol forces the client to check for the status of a certificate. This results in a lot of traffic for the CA behind the certificate.\nOCSP stapling moves the check to the owner of the certificate. On a regular basis the Nginx server will perform the check, receiving a new OCSP response. This response is stapled upon the SSL/TLS process with the user client. Due to this addition, OCSP stapling ensures the client that the owner is keeping their certificate up-to-date and is still valid.\nConfiguration All these snippets needs to be added below the virtual host. OCSP stapling is only useful when using SSL and is enabled (ssl on or listen ssl).\nEnable OCSP stapling and verification # Turn on stapling ssl_stapling on; # Enable verification ssl_stapling_verify on; Define certificate for OCSP stapling Next is defining a certificate. This step is optional when the full certificate chain was already provided with the ssl_certificate statement. In case just the certificate is being used (not the parts of your CA), then this statement is needed:\n# Define chained certificate (optional if already defined with ssl_certificate). ssl_trusted_certificate /etc/nginx/ssl/rootCA_plus_intermediates_chained.crt; Define nginx resolving To ensure proper resolving used when querying the verification systems, define what resolvers and their cache time. We use a low timeout, to quickly move to the next resolver if the first one fails.\n# Define resolvers, with a cache time of 10 minutes. Also define timeout for resolving, to limit timeout length. resolver 1.1.1.1 9.9.9.9 valid=10m; resolver_timeout 3s; Notes You need at least nginx 1.3.7 for OCSP stapling to work.\nRun nginx -t to test your configuration, before reloading.\nsystemctl reload nginx.service\n","permalink":"https://linux-audit.com/web/securing-nginx-configurations-implementing-ocsp-stapling/","tags":["certificates","cryptography","nginx","ssl","tls","web"],"title":"Securing nginx configurations: implementing OCSP stapling"},{"categories":["Hardening","Secure"],"contents":"It is still common that people do not know where to start when it comes to information security. With 5 basic principles we can improve the Linux system security and question ourselves if we have done enough.\n1. Know your system(s) The first principle is about knowing what your system is supposed to do. What is its primary role, what software packages does it need and who needs access?\nBy knowing the role of the system you can better defend it against known and unknown threats.\nSecurity Measures: Password policy Proper software patch management Configuration management Documentation 2. Least Amount of Privilege Each process running, or package installed, might become a target. Security professionals call this the \u0026ldquo;attack surface\u0026rdquo;. What you want is to minimize this attack surface by removing unneeded components, limit access and by default use a \u0026ldquo;deny unless\u0026rdquo; strategy. This latter means that access by default is blocked, unless you allow it (whitelisting).\nSecurity Measures: Use minimal/basic installation Only allow access to people who really need it 3. Perform Defense in Depth Protect the system by applying several layers of security. This principle is named \u0026ldquo;defense in depth\u0026rdquo; and can be compared with an onion: to get to the core, you have to peel of layer by layer. One broken defense might help us protect against full compromise.\nSecurity Measures: IPtables / Nftables Hardening of software components 4. Protection is Key, Detection is a Must Security focuses on the protection of assets. While this is a primary objective, we should consider that one day our defenses are broken. Therefore we want to know this as soon as possible, so we can properly act. This is where principle 3 and 4 both are linked. Set-up proper detection methods, similar to the trip wires used by the military.\nSecurity Measures: Linux audit framework Remote Logging Create backups and test them 5. Know your Enemy You can only protect a system the right way, if you know what threats you are facing. Why would this system be a target and who would be targeting it? Perform a risk analysis and determine what potential threats your system might endure.\nSecurity Measures: Vulnerability scans Penetration tests Risk analysis ","permalink":"https://linux-audit.com/5-basic-principles-of-linux-system-security/","tags":["documentation","linux","security"],"title":"5 Basic Principles of Linux System Security"},{"categories":["Compliance","Malware","PCI DSS compliance"],"contents":"An important part in the PCI DSS compliance, is checking for malicious software, or malware. By using anti-virus software like ClamAV, malware threats can be detected, and in most cases prevented. In this article we focus mainly on Linux environments, but of course most of these tips will apply to other platforms like Mac OS.\n5.1. Verify presence of software 5.1 For a sample of system components including all operating system types commonly affected by malicious software, verify that anti-virus software is deployed if applicable anti-virus technology exists.\nFirst thing is actually determining if ClamAV is installed on the system. This can be done by querying the related package manager on Linux:\ndpkg -l clamav rpm -q clamav pacman -Q clamav 5.1.1 Review documentation 5.1.1 Review vendor documentation and examine anti-virus configurations to verify that anti-virus programs;\n- Detect all known types of malicious software,\n- Remove all known types of malicious software, and\n- Protect against all known types of malicious software.\nThe documentation of ClamAV can be found on its website . It can detect every common form of malware. Depending on the mode the software is used, it can prevent, detect or remove the related files.\nClamAV can be used in several ways:\nDaemon mode\nProcesses can interface with the Clam daemon and request a file descriptor to be tested. The daemon then will return if the file is OK, or a different code (infected, no permissions etc).\nManual scanner\nThe clamscan utility can be used to manually check files. This is also a great way of running a regular scan on Linux machines, by using clamscan in a cronjob.\n5.1.2 Monitoring of malware threats 5.1.2 Interview personnel to verify that evolving malware\nthreats are monitored and evaluated for systems not currently\nconsidered to be commonly affected by malicious software, in\norder to confirm whether such systems continue to not require\nanti-virus software.\nThis is a procedural item to test by interviewing the technical contact persons.\n5.2 Management of anti-virus mechanisms 5.2 Ensure that all anti-virus mechanisms are maintained as follows:\n- Are kept current,\n- Perform periodic scans\n- Generate audit logs which are retained per PCI DSS Requirement 10.7.\n5.2.a Policies and procedures for anti-virus definitions 5.2.a Examine policies and procedures to verify that anti-virus software and definitions are required to be kept up to date.\nThis is a non-technical item and should be checked in the documentation of the organization.\n5.2.b ClamAV configuration \u0026ldquo;5.2.b Examine anti-virus configurations, including the master installation of the software to verify anti-virus mechanisms are:\n- Configured to perform automatic updates, and\n- Configured to perform periodic scans.\nTo keep ClamAV up-to-date, the freshclam utility can be used. Additionally monitoring can be set to check that signatures are regularly updated. One option is to use the clamconf utility and determine the date of the signatures.\n5.2.c Proper functioning of ClamAV 5.2.c Examine a sample of system components, including all operating system types commonly affected by malicious software, to verify that:\n- The anti-virus software and definitions are current.\n- Periodic scans are performed.\nTo ensure this item of the PCI DSS compliance verification is completely done, perform the following steps.\nFreshclam\nDetermine how freshclam is running (freshclam in daemon mode or manual). Additionally check if it is properly logging to /var/log/clamav/freshclam.log. Determine if freshclam encountered any issues, like outdated definitions.\nClamd and clamscan\nCheck if clamscan is scheduled via a cronjob. Additionally check if clamd is running and available for other software components to use it (e.g. mailbox scanning via MTA).\n5.2.d ClamAV logging 5.2.d Examine anti-virus configurations, including the master installation of the software and a sample of system components, to verify that:\n- Anti-virus software log generation is enabled, and\n- Logs are retained in accordance with PCI DSS Requirement 10.7.\nCheck the log files in /var/log/clamav and determine if the software is properly running.\nThe configuration files (/etc/clamav/clamd.conf and /etc/clamav/freshclam.conf) should have a log file defined. Use the clamconf utility to quickly determine if logging is enabled:\n# clamconf | grep log LogFile = \u0026#34;/var/log/clamav/clamd.log\u0026#34; LogSyslog disabled DevLiblog disabled LogSyslog disabled UpdateLogFile = \u0026#34;/var/log/clamav/freshclam.log\u0026#34; Example output:\nClamconf utility shows all settings of clamd, clamscan and freshclam\n5.3.a ClamAV configuration and status 5.3.a Examine anti-virus configurations, including the master installation of the software and a sample of system components, to verify the anti-virus software is actively running.\nCheck the process listing to see if ClamAV is running. If the ClamAV daemon (clamd) the process should show up. It is common to see also the freshclam utility showing up, when using it as a daemon.\n5.3.b Protection of ClamAV configuration files 5.3.b Examine anti-virus configurations, including the master\ninstallation of the software and a sample of system\ncomponents, to verify that the anti-virus software cannot be\ndisabled or altered by users.\nCheck the file permissions of the following files:\n/etc/clamav/clamd.conf /etc/clamav/freshclam.conf The files should be owned by the root user. Only root should be able to change the configuration file.\n5.3.c Procedures regarding ClamAV management 5.3.c Interview responsible personnel and observe processes to\nverify that anti-virus software cannot be disabled or altered by\nusers, unless specifically authorized by management on a\ncase-by-case basis for a limited time period.\nThis is a procedural test.\n5.4 Documentation regarding malware protection 5.4 ****Examine documentation and interview personnel to verify that security policies and operational procedures for protecting systems against malware are:\n- Documented,\n- In use, and\n- Known to all affected parties.\nThis is also procedural and needs manual testing, by interviewing.\nCommon findings As can be seen, PCI DSS is extensive when it comes to testing anti-virus and malware solutions. Some common findings on Linux systems include the lack of anti-virus, outdated virus definitions, or not properly configured.\nEmpty database\nUse monitoring to determine if signatures are loaded.\nDatabase information ------- Database directory: /var/lib/clamav Total number of signatures: 0 Signatures outdated\nAnother common issue is that the signature files are not up-to-date. This can be easily tested by reviewing log files, or feeding them into your SIEM.\nClamAV signatures outdated\nClamscan not functioning\nTo check if ClamAV is properly working, download the EICAR test file and run clamscan.\nTesting ClamAV with EICAR test file\n","permalink":"https://linux-audit.com/using-clamav-for-linux-pci-dss-requirement-5-malware-and-anti-virus/","tags":["clamav","malware","pci dss"],"title":"Using ClamAV for Linux PCI DSS requirement 5: Malware"},{"categories":["Auditing","Linux","Passwords"],"contents":"Linux systems use a password file to store accounts, commonly available as /etc/passwd . For additional safety measures, a shadow copy of this file is used which includes the passwords of your users. Or actually hashed password, for maximum security. This shadow file is /etc/shadow and has line like this:\nusername:$6$6Y/fI1nx$zQJj6AH9asTNfhxV7NoVgxByJyE.rVKK6tKXiOGNCfWBsrTGY7wtC6Cep6co9eVNkRFrpK6koXs1NU3AZQF8v/:16092:0:99999:7::: For proper display, let\u0026rsquo;s split this up in several fields:\nusername $6$6Y/fI1nx$zQJj6AH9asTNfhxV7NoVgxByJyE.rVKK6tK 16092 0 99999 7 empty empty Field explanations Time to have a look what all these strings mean:\n1) Username The first field is an easy one, it is the username of the particular account.\n2) Password hashing details + hashed password The most important string in the /etc/shadow file is definitely the second field. It includes the password details and consists of several parts. In this case it starts with \u0026lsquo;$6$\u0026rsquo;, which refers to SHA-512.\nNote: if the password field just has a ! or *, then the account is locked. A double ! (!!) indicates a password has never been set.\nSchemes overview Identifier Scheme 1 md5crypt 2, 2a, 2b, 2x, 2y bcrypt 3 NTHASH 5 sha256crypt 6 sha512crypt 7 scrypt 8 PBKDF2 with different implementations gy gost-yescrypt md5 Solaris MD5 sha1 PBKDF1 with SHA-1 y yescrypt MD5 Indicated with \u0026lsquo;1\u0026rsquo;, usually MD5 with 22 characters.\nSHA-256 Indicated with \u0026lsquo;5\u0026rsquo;, usually SHA-256 with 43 characters.\nSHA-512 Indicated with \u0026lsquo;6\u0026rsquo;.\nSecond part is salt and separators (in this case $6Y/fI1nx$). The salt is a small string of characters to mix into the hashing function. Its goal is making it more difficult to perform certain password based attacks on the hashed password. This salt consists of characters a-z, A-Z, 0-9, / and .\nLong string of characters = hashed password\nThe long string and its length depends on the hashing method used. With $6, or SHA-512, it will 86 characters.\nYescrypt Indicated with a \u0026lsquo;y\u0026rsquo;. It has a number of rounds between 1-11, with the default being 5 rounds.\nTo see the number of rounds, look at the second string. For example: j9T=5 rounds, jAT=6 rounds, jBT=7 rounds, jCT=8 rounds, jDT=9 rounds, jET=10 rounds, jFT=11 rounds.\n3) Last changed This number indicates when the password was last changed. The number does indicate the day number, starting from epoch date (1 January 1970). Right now that is in the 16000+ range.\n4) Number of days before password can be changed This field defines how long it takes before the password can be changed. In our case zero, so it can be changed now.\n5) Number of days till required password change Another pretty self-explanatory field, stating how long is left (in days), before a password change is required. A great option to force password changes.\n6) Warning threshold in days In line with previous field it describes the number of days till a warning will be giving. In this example it is a week.\n7) Expire date Also stored in days, describing when the account was expired (from epoch date).\n8) Reserved field Usually not used by Linux distributions.\nFile Permissions The /etc/shadow file should be owned by the root user, with usually shadow as group owner. This file should not be world-readable, therefore 640 or 400 would be an appropriate file permission.\nTips for reviewing the /etc/shadow file By default the shadow file looks cryptic (pun intended). To simplify the output, one could use getent combined with awk and column It extracts all entries from the shadow database, extracts entries where the second field starts with a dollar sign ($), and finally presents it in a more friendly way.\ngetent shadow | awk -F: \u0026#39;$2 ~ /^\\$/\u0026#39; | column --table --separator :$ Consistency checking of /etc/passwd and /etc/shadow Linux distributions usually provide a pwck utility. This small utility will check the consistency of both files and state any specific issues. By specifying the -r it may run in read-only mode.\nExample when running pwck on /etc/passwd and /etc/shadow file\nAnything missing from this article? Let it know!\n","permalink":"https://linux-audit.com/authentication/password-security-with-linux-etc-shadow-file/","tags":["etc","linux","passwd","password","pwck","shadow"],"title":"Password Security with Linux /etc/shadow file"},{"categories":["System Administration"],"contents":"Using pkg_add Keeping your systems stable and secure\nEvery system needs to stay up-to-date with its packages, including OpenBSD. Most OpenBSD users already use pkg_add for the installation of packages. This utility can also be used for package upgrades.\nOption 1: Use /etc/installurl Newer OpenBSD versions use the file /etc/installurl to select the mirror for pkg_add.\nOption 2: PKG_PATH The first thing to do is defining your PKG_PATH. This will usually be the address of a FTP or HTTP server, which has the latest packages available. To have this variable set every time you log in, use the file .profile in your home directory (e.g. /root/.profile). Add the full export line below:\nexport PKG_PATH=\u0026ldquo;http://ftp.openbsd.org/pub/OpenBSD/$(uname -r)/packages/$(arch -s)/\u0026rdquo;\nSince it won\u0026rsquo;t be active the first time you added it, execute the command, so the variable PKG_PATH will be set.\nTip: use a mirror to improve your performance and reduce load on the main systems.\npkg_add Next step is the actual execution of pkg_add. To upgrade us the -u, use the -i for interactive and -v for verbose output.\n# pkg_add -uvi\nUpdate candidates: quirks-2.9 -\u0026gt; quirks-2.9 (ok)\nquirks-2.9 signed on 2014-07-31T22:37:55Z\nUpdate candidates: gettext-0.19.1p0 -\u0026gt; gettext-0.19.1p0 (ok)\nUpdate candidates: libiconv-1.14p1 -\u0026gt; libiconv-1.14p1 (ok)\nUpdate candidates: libidn-1.28p0 -\u0026gt; libidn-1.28p0 (ok)\nUpdate candidates: p5-Clone-0.36p0 -\u0026gt; p5-Clone-0.36p0 (ok)\nUpdate candidates: p5-Curses-1.28p3 -\u0026gt; p5-Curses-1.28p3 (ok)\nUpdate candidates: p5-Curses-UI-0.9609 -\u0026gt; p5-Curses-UI-0.9609 (ok)\nUpdate candidates: p5-DBD-SQLite-1.35p3v0 -\u0026gt; p5-DBD-SQLite-1.35p3v0 (ok)\nUpdate candidates: p5-DBI-1.631p0 -\u0026gt; p5-DBI-1.631p0 (ok)\nUpdate candidates: p5-FreezeThaw-0.5001 -\u0026gt; p5-FreezeThaw-0.5001 (ok)\nUpdate candidates: p5-MLDBM-2.05 -\u0026gt; p5-MLDBM-2.05 (ok)\nUpdate candidates: p5-Net-Daemon-0.48 -\u0026gt; p5-Net-Daemon-0.48 (ok)\nUpdate candidates: p5-Params-Util-1.07p0 -\u0026gt; p5-Params-Util-1.07p0 (ok)\nUpdate candidates: p5-PlRPC-0.2018p1 -\u0026gt; p5-PlRPC-0.2018p1 (ok)\nUpdate candidates: p5-SQL-Statement-1.405 -\u0026gt; p5-SQL-Statement-1.405 (ok)\nUpdate candidates: p5-Term-ReadKey-2.30p6 -\u0026gt; p5-Term-ReadKey-2.30p6 (ok)\nUpdate candidates: pcre-8.35 -\u0026gt; pcre-8.35 (ok)\nUpdate candidates: pkg_mgr-0.2.1p2 -\u0026gt; pkg_mgr-0.2.1p2 (ok)\nUpdate candidates: sqlports-compact-4.1 -\u0026gt; sqlports-compact-4.1 (ok)\nUpdate candidates: wget-1.15 -\u0026gt; wget-1.15 (ok)\nWhen updates are available you will be prompted to install it. In this case, our system was up-to-date, which results in the \u0026ldquo;(ok)\u0026rdquo; behind each package.\n","permalink":"https://linux-audit.com/updating-all-openbsd-packages-with-pkg_add/","tags":["packages","software"],"title":"Updating all OpenBSD packages with pkg_add"},{"categories":["Software","System Administration"],"contents":"Maximum Linux security with proper software patch management\nSoftware upgrades are almost as old as the first lines of software code. Still companies struggle to properly update software, also when it comes to security patching. In this article we have a look at the reason behind patching and some methods to keep your systems humming, with fresh packages.\nWhy Update? To most of us, it instantly makes sense to keep the software on your systems up-to-date. But still, sometimes we have to explain others why we do actually need a process to apply software updates and patches.\nThe most important reasons to keep software up-to-date are:\nFixes to known software bugs Enhancements (new features or small improvements) Solve security issues and vulnerabilities Linux Software Each Linux based system is running the Linux kernel, together with supporting tools. These tools are collected by the distribution you are running. It is the vendor of this distribution which is responsible for tracking software updates. Depending on how \u0026ldquo;aggressive\u0026rdquo; the Linux distribution is, it may include many packages and updates, or be more conservative.\nBleeding Edge VS Stable\nArch Linux is one in the first group, being \u0026ldquo;bleeding edge\u0026rdquo;. It is much more progressive to include newer software packages. Red Hat on the other hand wants their customers to run very stable machines. Therefore they won\u0026rsquo;t add new features to an existing version, however will solve major bugs and security vulnerabilities. The result is older versions of the same software packages. Both have their advantages and clearly their disadvantages. In any case, all distributions have to track security updates, so they can provide patches.\nManagement Tools For Linux patch management there are several different tools available. In categories they would be:\nManual Patching Shell Scripts Automation \u0026amp; Configuration Tools Software Management Tools Manual Patching The most basic form of security patching is by hand. Simply running commands like apt-get update and yum update. The clear disadvantage is that this form is the most time consuming. This form of patching is prone to mistakes, as each system is different, especially with a stacked up amount of patches.\nShell Scripts The next level of patching is using small scripts. It could be a small script, which automates several steps of the patching process. By distributing the same script to multiple machines, it ensures some level of repeatability, which equals quality.\nAutomation \u0026amp; Configuration tools Tools like Ansible, Chef or Puppet can help with automating the security patch management. Simply feed the tool an update command, or what minimum version a software package should be.\nSoftware management tools Last but not least, most Linux vendors provide their own solution for managing software packages. It shows the installed packages and marks what packages have a security bulletin attached. Ubuntu has Landscape product, Red Hat has Satellite (see screenshot below).\nPatch management solution from Red Hat\nSecurity updates Depending on your situation, you might want to subscribe to a security related mailing list. This might be a generic one tracking software on Linux, or even a distribution specific list for security. Even if you have your patch management automated, you might want to perform regular audits. A small adjustment in your network configuration might result in non-patched systems, which with the right tool can be easily detected.\nAutomatic updating Most package managers support automatic updating, or have supporting tools available to make this possible. For example the unattended-upgrades package helps to keep Debian/Ubuntu up-to-date. It is even possible to define what repositories should be used, so you give it a \u0026ldquo;security updates only\u0026rdquo; policy.\n","permalink":"https://linux-audit.com/software/software-patch-management-for-maximum-linux-security/","tags":["patch management","software","software management"],"title":"Software Patch Management for Maximum Linux Security"},{"categories":["Auditing","Compliance","PCI DSS compliance"],"contents":"Some areas are within the PCI DSS standard are definitely not directly clear when reading the description. Section 10.2.7 is one of them. It talks about the creation and deletion of system-level objects and specifically the ability to log them.\nSystem-level objects? The guidance in 10.2.7 speaks about malware and mentions database related items. That does not make auditing very obvious, as malware usually targets binaries. Therefore we have to look first what a system-level object is.\nLet\u0026rsquo;s start with a possible definition:\nSystem-level object:\nAnything on a system which is required for its operation, including the kernel, executables/binaries, configuration files, libraries, drivers and software applications.\nFor Linux and Unix based systems this equals to directories like /bin, /sbin, /etc/, /var/lib etc. In this article we will set-up auditing on the most important system objects of Linux system.\nMonitoring system-level objects To protect important areas of the Linux system, proper file permissions will help to safeguard it against unauthorized change or deletion. Defending is for PCI not enough, as we also need proof that accounting is in place. For that we can use the Linux audit framework.\nAll rules shared in this article can be easily tested by adding them with auditctl, followed by the rule itself. Rules start with -a, to indicate they should be added to the existing rule base. By defining a -d as first parameter, we can delete the related rule again.\nAuditing Linux kernel First area to audit is the Linux kernel itself. Every alteration, including upgrades, should be properly prepared, installed and finally be logged. With the audit daemon we can monitor any changes to the kernel files itself. As we want to keep an eye on the boot configuration as well, monitoring /boot is a great start.\n-a always,exit -S all -F dir=/boot -F perm=aw -k system-objects\nAuditing system binaries Next area is auditing the system binaries. There are divided over several directories. You have to look at your system where binaries are listed. Usually displaying the $PATH variable will give a good indication.\n-a always,exit -S all -F dir=/bin -F perm=aw -k system-objects\n-a always,exit -S all -F dir=/sbin -F perm=aw -k system-objects\n-a always,exit -S all -F dir=/usr/bin -F perm=aw -k system-objects\n-a always,exit -S all -F dir=/usr/local/bin -F perm=aw -k system-objects\n-a always,exit -S all -F dir=/usr/local/sbin -F perm=aw -k system-objects\n-a always,exit -S all -F dir=/usr/sbin -F perm=aw -k system-objects\nNote: Check your system if there are any other directories containing binaries and not being a subdirectory of any of these entries above.\nAuditing system libraries Last are the system libraries. They contain system calls and supporting functionality for most of the system binaries.\n-a always,exit -S all -F dir=/lib -F perm=aw -k system-objects -a always,exit -S all -F dir=/lib64 -F perm=aw -k system-objects -a always,exit -S all -F dir=/usr/lib -F perm=aw -k system-objects Some distributions mights use /usr/lib/. As we already have a watch on the /usr/lib, these should be included (because of the usage of dir).\nAuditing configuration files Most of the configuration files are stored in /etc. If you have other locations, add them as well.\n-a always,exit -S all -F dir=/etc -F perm=aw -k system-objects Auditing systemd If your system is using systemd, don\u0026rsquo;t forget to audit the related directories as well.\n-a always,exit -S all -F dir=/usr/lib/systemd/ -F perm=aw -k system-objects Note: If you are already auditing /usr/lib with the -F dir=, the systemd directory will be included.\nMonitoring databases As the guidance provides hints to databases, we suggest to monitor the related database directories as well. The creation of deletion of a database might be useful to log. However, keep in mind on what is actually being logged and not to have too much accounting data, as database can be change very regularly.\nAnother useful thing is to consult the documentation of the database software, to ensure stored procedures and changes to tables are logged.\nNotes The defined rules with permission \u0026ldquo;aw\u0026rdquo; will be triggered when an item is created, updated or deleted from the specified directories. The \u0026ldquo;a\u0026rdquo; permission is responsible for monitoring the creation and deletion, as these involve the change of the attributes of the file. While writing, the attributes (e.g. mtime) will change as well. The writing to the file is monitored with the \u0026ldquo;w\u0026rdquo; permission definition.\nSearch events After an event is logged, it can be discovered with the ausearch utility. Simply provide the -k parameter, followed by the earlier defined name \u0026ldquo;system-objects\u0026rdquo;.\nAutomatic auditing with Lynis To ensure your systems are PCI compliant, use supporting tools like Lynis, to check your PCI compliance on a regular basis. This way changes can be discovered quickly, avoiding a failure at the next audit.\nHappy auditing!\n","permalink":"https://linux-audit.com/compliance/pci-dss-v3-linux-creation-and-deletion-of-system-level-objects-10-2-7/","tags":["auditd","compliance","linux","monitoring","pci dss"],"title":"PCI DSS Linux: Creation and deletion of system-level objects"},{"categories":["Auditing","Lynis"],"contents":"Lynis 2.x will bring security auditing of Linux and Unix systems to a new level. In this blog post we share some exciting new features.\nRelease of Lynis 2 is planned for February 2015.\nOverview:\nHistory Lynis 2.x Plugins Systemd Support File Integrity Monitoring Containers \u0026amp; Virtualization Operating Systems Focus on Simplicity Free and Commercial Support History Lynis has been created in 2007, as a follow-up on the well-known tool Rootkit Hunter (rkhunter). Both tools are now used by companies all over the world, from individuals up to big companies, military and governments.\nWhere Rootkit Hunter only searched for malware, Lynis was about scanning systems broader and deeper. The goal was simply measuring the security defenses of a particular system and assisting with hardening.\nSince 2013, Lynis under the support of CISOfy , to ensure ongoing development and keep up with new trends. With the author still being involved in development and promotion, the tool gained much traction over the last years. This resulted in more downloads and reaching even more users than ever before. The tool has been covered on many blogs, magazines and recently ended 3rd in ToolsWatch\u0026rsquo;s hunt for the best security tool of 2014.\nLynis 2.x Now seven years later, the goal is much higher. Instead of just hardening, the new Lynis wants to help you preventing break-ins, detecting intruders and provide continuously security monitoring. In this post we share some of the upcoming developments.\nPlugins The biggest change in Lynis 2.x might the usage of plugins. It implies a difference in thinking about how individual tests are performed. In the past different types of tests where placed in the category file (e.g. include/tests_firewalls). From now on, they are split into two different types:\nInformation gathering Configuration checks Information gathering\nThe first type of tests simply gather information on the system, or in other words system discovery. Examples include a process listing, files in a directory, or the status of a particular program.\nConfiguration checks\nConfiguration checks on the other hand, are the ones who you might recognize from the screen output. They actually make an informed decision on what has been found and tell the user the result. This may include actual advice or a suggestion.\nMost plugins will focus on the information gathering. Where appropriate, it might be an input for configuration checks and provide a suggestion here and there. With this new separation, the scan process will be more efficient, resulting in quicker scans.\nThe plan is to release both community plugins and commercial plugins. The last group will focus on the Enterprise functionality, like compliance and file integrity monitoring.\nSystemd support While there are many strong opinions about systemd, and its strengths and weaknesses, the truth is that it is out there. Since systemd is still under (heavy) development, we will be the first auditing tool to do a deep analysis of systemd, and keep up with its development. Each related Lynis test will determine what systemd options are available and process the related output, helping you to understand systemd more easily.\nAuditing is underrated?\nWe believe that auditing is still underrated by companies. After all, even with a lot of auditing in place, systems get compromised. That does not mean auditing is useless. Too often simple basic details are being overlooked, which later then turn out to be cause for a break-in.\nWhile Systemd enhances the boot process, it might complicate things. Proper auditing is needed to keep it well configured. Analyzing systemd is something which is of great value to maintain a healthy system. Our goal with Lynis 2.x is to audit it inside out and tell you when things can be improved.\nRelated Tests:\nDetermine available units Checked for failed units Analyze coredump configuration Determine timers Check journal and its configuration File Integrity Monitoring Being a host based auditing utility, also indicates that the files on the system can be analyzed. During the audit much information can be gathered about what security defenses are applied, and how many. One interesting area is that of determining the integrity of files, packages and data. Lynis 2.x will focus even more on these areas.\nSome examples which will be tested in the newer Lynis versions:\nGPG signing Vulnerability database Aging of files Presence of file integrity monitoring tools Additionally a plugin will be released which will do more data collection, so this data can be stored and compared.\nContainers \u0026amp; Virtualization The upcoming releases of Lynis have improved detection for both virtualization and container technology. By better determination of the environment a machine is running it, the better advice can be provided.\nControl groups (cgroups) and Namespaces\nNot new, but definitely more used in the last years are cgroups and namespaces. Both combined they prioritize, control and restrict processes. When used properly, powerful protection against limited resources. From a security point of view they are both very interesting options.\nDocker A few posts have already been dedicated to this subject. What is new is following how container technology and the role of Docker will develop. Lynis will be tracking these developments to ensure the security aspects. With containers still not being able to fully contain, companies and individuals might be sharing way too much they intended. Lynis will help detecting possible gaps.\nSecure Dockerfile(s)\nOne of the first areas to secure Docker is building your images securely. With the new command lynis audit dockerfile, you can test your Dockerfile for security best practices.\nUbuntu Snappy Another company, known for its Linux distribution Ubuntu, is Canonical with its product Snappy. Definitely something which will be used by Ubuntu users in the (near) future. Enough reason to ensure those areas are audited.\nOperating Systems Lynis has been tested on many different platforms and versions. From Solaris to lesser known Linux distributions, the tool runs on all of them.\nPortable code\nThe big difference with benchmarks and other tools, is that it uses a \u0026ldquo;discover on the go\u0026rdquo; method. So it actually learns per machine what can be tested, how much is needed and what tools are available. This method has proven to be successful and one of the reasons why tools like OpenSCAP simply don\u0026rsquo;t take off. Most of you do not want to run binaries, have to compile things manually or have to read hundreds of unclear findings.\nTo ensure portability of our code, we regularly test it on different platforms and perform an extensive analysis on the related log file. Within Lynis 2.x we will continue this effort and focus on the key differences of each operating system.\nArch Linux One of the great benefits of Linux is that it is open source and new features in software are quickly shared. Arch Linux being a \u0026ldquo;rolling release\u0026rdquo; distribution, means it is always up-to-date and using new great features. With the Lynis 2 releases, Arch Linux will be better supported. As they are one of the pioneers when it comes to new features, Lynis will keep an eye on them.\nGentoo Gentoo and clones are still being used a lot. With the great support of Gentoo to include Lynis, we will keep on supporting this Linux distribution in the best possible way.\nRed Hat One of the giants in open source is Red Hat and their commercial product line. For those using clones like CentOS, Scientific Linux or Red Hat\u0026rsquo;s playground distribution Fedora, Lynis will leverage common used technologies. One of best known parts might be that of the Linux Audit framework.\nSuSE SLES Commercially available operating systems are interesting for corporate users. With the upcoming Lynis releases we will add additional support for commercial platforms like those of SuSE. This will include the detection of specific tools not found on other distributions.\nUbuntu Originally Ubuntu was seen as a desktop distribution. During the last years it has become clear Ubuntu is also suitable for servers. With upcoming container technologies, Ubuntu will implement Snappy. Lynis will aim for including as many technologies as possible. By proper segmentation, audits will remain quick and at the same time the code base will be as simple as before.\nFree and Commercial Support Too many open source projects became closed source or abandoned after an organization got involved with its development. We are so passionate about our software and the community of users, that we want to keep Lynis open source and available to the community.\nThis means that the Lynis remains free and can be used as a standalone tool. At the same time it is part of the Lynis Enterprise solution, being the client which collects data and brings it to the central node.\nSupport\nGood software needs the right support. If your business is relying on security tools like Lynis, you will benefit from having an up-to-date tool. To ensure you have the best possible tools in your toolbox, we continue to release often. Free users will benefit from the development efforts by the supporting company, while customers will benefit from a well-known tool, which is well-known and peer reviewed by the community.\nTransformation\nWith Lynis version 2, we want to finish the transformation from a hobby project, to a well-known and rock-solid solution for system administrator and security professionals.\nLynis Features:\nLicense: Open Source, GPLv3 Pricing: Free Primary goal: Security auditing and system hardening Developer: CISOfy Project URL: https://cisofy.com/lynis/ Lynis Enterprise Features:\nAdditional plugins (e.g. file integrity, compliance) Central management interface Reporting possibilities Automation, snippets and defensive hardening tools Project URL: https://cisofy.com/lynis-enterprise/ Do you believe in open source and security auditing? Share this post in your social network and get the internet a safer place.\n","permalink":"https://linux-audit.com/lynis/whats-new-lynis-2-features/","tags":["arch linux","gentoo","lynis","one-time","ubuntu"],"title":"What's New in Lynis 2: Features"},{"categories":["Linux"],"contents":"Background of Linux security modules Like normal kernel modules, security modules extend the basic functionality of the Linux kernel. The need for a modular structure was proposed when SELinux was being introduced. There was a little discussion to use modules or not, as SELinux was the only one being available. Some people proposed apply it as a kernel patch, but in the end Linux creator Torvalds, decided to make this type of functionality modular. The first security module was born.\nHow it works Linux security modules are relying on kernel hooks. These fixed pointers in the kernel, or kernel interface, allow an external component to influence the behavior of the kernel. The interesting part of these modules, is that they are restrictive in nature. This means they will lower the privileges someone, or some process, already might have. This is the opposite of tooling like sudo for example, where one actually acquires new privileges.\nCommon frameworks SELinux is one of the most known Linux security modules available. This framework uses the approach named MAC short for mandatory access control . MAC based systems use subjects and objects. Subjects are the \u0026ldquo;active\u0026rdquo; participant, like a user or process, where objects are the items to be accessed (e.g. a file). Together you can form a policy, which decides who can do (to) what.\nAppArmor Created by Immunix, AppArmor is a similar MAC based framework as SELinux. Immunix was acquired by Novell, resulting AppArmor to be found on SUSE Linux. AppArmor has been ported to others, like Debian, Gentoo and Ubuntu.\nA big difference between is in the way files (objects) are monitored. AppArmor monitors files by path, where SELinux does it by security labels.\nSome benefits over SELinux:\nConsidered to be easier in administration File system independent, which means no specific support within the file system is needed (security labels) Disadvantages:\nWhen creating a hardlink of a file, it may become accessible again (as the inode has changed) Module configuration Most Linux security modules can be installed as a package. Depending on the specific distribution you are using, it may be installed by default. For example SELinux is commonly found on Red Hat based systems, where AppArmor is available on SUSE Linux and Ubuntu.\nUsually a Linux security module is configured with its own configuration files, while being enabled or disabled via a sysctl value. Others are so small, that they can be tuned via just sysctl. For example the ptrace capabilities on processes via YAMA:\nkernel.yama.ptrace_scope = 1\nOverview by Year 1998\nAppArmor (then SubDomain), used in Immunix Linux\n2007\nIntroduction of SMACK\nAppArmor ported to Ubuntu Linux and AppArmor development taken over by Canonical.\n2009\nTOMOYO Linux\n","permalink":"https://linux-audit.com/introduction-into-linux-security-modules/","tags":["apparmor","linux","selinux"],"title":"An Introduction Into Linux Security Modules"},{"categories":["Accounting","Auditing","System Administration","Troubleshooting"],"contents":"Starting with Linux auditing can be overwhelming. Fortunately, there is a great feature in the Linux kernel to watch events and log them for us. To give you a quick start to use the Linux Audit Framework, we have collected some basic rules for configuring the audit daemon and its rules.\nMain Configuration By default the configuration values in /etc/audit/audit.conf are suitable for most systems. If you know your system is very low or very high (e.g. mainframe) on resources, then you might want to adjust some file sizes or buffers.\nAuditing does not equal security The auditing framework simply monitors and logs events to an auditing log. Keep in mind running an auditing daemon does not increase security in itself. It does however create an audit trail, helping with detection (e.g. security intrusion).\nRules The Audit daemon uses rules to monitor for specific items and create a related event log. Each rule can be provided to the daemon by using the configuration file /etc/audit/audit.rules or with the command line utility auditctl. When using the configuration file, keep in mind that just adding new rules is not enough to activate them. Reread of the configuration file is needed.\nThe rules file can be read with -R. It should be owned by the root user, or a \u0026ldquo;Error - /tmp/test isn\u0026rsquo;t owned by root\u0026rdquo; will show up.\nThe three types The auditing framework and daemon in particular, knows 3 types of rules:\nBasic auditing settings File and directory watches Syscall monitoring First match wins When a rule matches, the audit daemon stops evaluating if other rules need to be checked as well. So make sure to put things in the right order of processing, or some rules will never match.\nGetting the right details When using a watch on a directory, less information will be logged in comparison with specific file watches. So when in need of all details, monitor files instead.\nFile needs to exist When using templates or adding new file watches, keep in mind that the files or directories to monitor, need to exist on disk.\nSystem architecture Some rules might not work between systems, if their architecture is different. Where possible, specify the architecture to ensure you are monitoring the right system call.\nUse keys File access monitoring with Linux audit framework\nTo simplify searching and categorizing events, use keys. Multiple keys can be used on a rule, which help in grouping events while still having a separation in place as well.\nThe ausearch utility can be told to search by key with the -k parameter, followed by the actual key. This way searching for specific events becomes much easier.\nKeys also help in grouping the events. This way you can use it for both auditing purposes and use some specific key combinations for goals like intrusion detection. It minimizes the amount of events, which you might want to put into a SIEM (Security Incident and Event Management) solution.\nPrepare before auditing Carefully select which files or events you want to monitor. With more auditing, the load will increase. Additionally, the audit log will increase as well. Too much logging and you will be overwhelmed. The \u0026ldquo;log everything\u0026rdquo; approach is definitely not the right mindset when using Linux auditing capabilities.\nCheck the examples The audit package contains some great example files. Have a look at your system at the files: capp.rules, lspp.rules, nispom.rules, and stig.rules.\nConclusion With these rules, you should be able to get the Linux audit framework up and running. The audit framework is powerful for debugging and troubleshooting issues on your system. Additionally, it is of great help in detecting unauthorized events or system intrusion. If you like this subject, we encourage you to check out the other blog posts we have on this subject.\n","permalink":"https://linux-audit.com/linux-audit-framework/linux-audit-framework-101-basic-rules-for-configuration/","tags":["audit","auditctl","auditd","auditd.conf","ausearch"],"title":"Linux Audit Framework 101 – Basic Rules for Configuration"},{"categories":["Automation","Hardening"],"contents":"System administrators and engineers love to automate things. In the quest to get everything replaced by a script, automated hardening of systems is often requested. Unfortunately this automation might later backfire, resulting in a damaged trust in system hardening.\nWhy System Hardening? The act of increasing system defenses is a good practice. It helps protecting your valuable data, so it can only be used by authorized people. System hardening itself consists of minimizing services and removing unneeded ones. This also applies to the access to the system, by reducing the amount of users, network access and protocols. Last but not least, changing software configurations to include the encryption of data and add additional authentication layers.\nHardening Scripts More and more hardening solutions pop up, which promise to simplify hardening. Sure, system hardening is good, so is automation. But there is no \u0026ldquo;one fits all\u0026rdquo; solution when it comes to system hardening. Each system is different and needs a different level of protection. Your personal notebook might actually get a bad performance while browsing, if some network settings are adjusted by a Linux hardening script.\nAlternatives Normally I wouldn\u0026rsquo;t mind to name a few alternatives to our security auditing tool Lynis. In this case I feel strongly that promoting hardening scripts will actually weaken your security. You might end up taking a shortcut and end up with a false sense of security. Or worse..\nSecurity risks Some hardening scripts even download external files which they don\u0026rsquo;t control themselves. As hardening requires root permissions, this is definitely a serious risk. Automating your security controls is fine, but ensure you have 100% control over what is being automated. Another thing is properly testing, which might be hard if you don\u0026rsquo;t know what the tool is doing.\nThe Alternative = Auditing + Automation Instead of just automatically hardening Linux systems with a script, use a combination of auditing together with a configuration management tool like Puppet. This way it is easy to detect what might be improved, while at the same time apply automation.\nTailored security Sure, you might think that we would always advise to use an auditing tool, as we created one. But actually, it is free and open source. We honestly believe that measuring security and then acting on it appropriately, is the better way to deal with information security. Just running a hardening tool will definitely not give you the security level tailored to your needs, but it might give a false feeling of security.\nContinuous security monitoring When using the combination of auditing and automation, divide systems by category, customer, role or any custom attribute. Then give them the right security policy it deserves and finally measure again with the auditing tool.\nThis way of working is also often referred to as the PDCA cycle (plan, do, check, act), providing continuous auditing and monitoring.\nBy using the right combination of testing, researching, applying and testing again, you will enforce your security defenses more appropriately.\nKnow Your Hardening Last but not least, we didn\u0026rsquo;t go into the importance of knowing what you harden and why. For example changing kernel settings, or installing a firewall, might need specific knowledge. What is the point of applying hardening when some settings are not even applicable? Or adding firewall rules, while the firewall itself is not even running?\nEach security control requires some knowledge about the subject. That\u0026rsquo;s why we provided our tool, to first detect what might be improved, secondly providing the related background information. Then your expertise of your environment comes into play, where you can determine what controls are appropriate. A ready-to-use Linux hardening script will never beat that.\nHappy hardening!\n","permalink":"https://linux-audit.com/system-hardening/why-linux-security-hardening-scripts-might-backfire/","tags":["automation","hardening"],"title":"Why Linux security hardening scripts might backfire"},{"categories":["Containers"],"contents":"Everything you need to know about Docker security.\nIntroduction into Docker Docker became very popular in a matter of just a few years. Operating systems like CoreOS used Docker to power the system by running applications on top of their own lightweight platform. Docker in its turn, provides utilities around technologies like Linux container technology (e.g. LXC, systemd-nspawn, libvirt). Previously Docker could be described as the \u0026ldquo;automated LXC\u0026rdquo;, now it\u0026rsquo;s actually even more powerful. What it definitely is, is simplifying and enhancing the possibilities of container technology.\nContinuous delivery Rolling out containers is quick and very easy. It helps companies to improve development work flows. Developers can perform testing and deploy their applications much easier than before. Additionally, the use of Docker enhances the process from development, up to running software in production. This is achieved by using smaller units, which are easier to create, monitor and secure.\nSupporting multiple technologies Docker uses the possibilities of the hypervisor management tooling like libvirt and systemd-nspawn. It has an ongoing development and supports more and more features, to simplify the management of containers. With each of the component getting more stable, the base of Docker reached a level of stability, use it in production.\nEnhancing security With the right measures, Docker will also enhance security. For example due to running (smaller) individual units, controlling them is easier. One benefit of small containers is providing application owners and administrators a better insight what software, protocols and network flows are needed for individual services.\nIT architects and security professionals will definitely benefit from container technology as well. Architects gain fine-grained building blocks, to define new services and enhance existing ones. Security professionals benefit from a better segmentation and minimizing the permissions needed in each individual container.\nDocker Security and Risks Software packages can solve existing security threats, or actually introduce new risks. This is also the case when using Docker. While it can help in reducing risks by using compartmentalization, the implementation might have its flaws.\nRisks One common threat of new services is a low(er) stability, which forms a risk to the availability of a service. Another one is information disclosure, as the service might be lacking appropriate controls. Usually new technologies have a need of adding new features quickly, which might result in sloppy programming. Often this will result in software vulnerabilities, including ones which are security related.\nUnfortunately, Docker already had its share of security vulnerabilities, but they took a more active stance to improve the security of their products.\nMethods and best practices To reduce the risks when using fairly new technologies, we will have a look at the methods available to Docker. In particular, we look how Docker can increase security. After that, we provide some best practices when dealing with Docker containers.\nMaturity level Container management is a fairly new technology, which leaves many professionals with a knowledge gap in this area. Additionally, not many people are capable (yet) of making a proper security assessment. For example, auditors might not be able to ask the right questions regarding the implementation of containers.\nThis mature level risk also includes lacking technical auditing of containers, like difficulties in maintaining it and keeping documentation up-to-date. After all, the flexibility containers provide also means containers can run more easily on a different system, at any given time. This might need another level of documentation, to reflect the possibilities of each individual unit.\nContainers do not (fully) contain yet While containers are used to compartmentalize and limit resources, they actually don\u0026rsquo;t fully contain. For example, a process running in the container with UID 1000, will have the privileges of UID 1000 on the underlying host as well. Along these lines, a process running as root (UID 0) in a container has root-level privileges on the underlying host when interacting with the kernel.\nThis risk will be soon be mitigated when user namespaces are fully being implemented. The first step is already made in the form of subordinate user IDs (subuid). This helps mapping existing user IDs on the host system, into different user IDs within each container.\nSecurity Benefits of Docker Segregation of applications Normally applications all run on the same host system. By using container technology we can segregate them, making it easier to determining traffic flows.\nFlexible attitude With containers being smaller individual units, they become dynamic, or flexible. The work flow to maintain them is more flexible as well. Great for security patching, testing and releasing the updated containers into production.\nFocus on automation Docker has a clear focus on automation. They have supporting products like Docker Swarm (clustering) and Compose, to simplify management of many containers.\nLimiting information disclosure Containers can limited resources assigned. This helps us in limiting the amount of information available to the system (and an evil attacker). Each container gets the following components:\nNetwork stack Process space Filesystem instances Limiting resources are achieved by using namespaces. Namespaces are like a \u0026ldquo;view\u0026rdquo;, which only shows a subset of all resources on the system. This provides a form of isolation: processes running within a container cannot see or affect processes in other containers, or the host system itself.\nProtection Methods By using Docker properly, some of its defenses can be used. Unfortunately, the tooling does not actively help yet to leverage all possibilities. You are the one to properly configure, use and update Docker. Our hope is that Docker in the future will be more strict with some options, or at least advise the user to some extent.\nLimited capabilities Linux has support for \u0026ldquo;capabilities\u0026rdquo;, which can be seen as roles. A role could be opening a network socket, to craft a packet and put it onto the wire. Normally these kind of roles are only available to the root user. By splitting them in the form of capabilities, they can also be assigned to individual processes as well. This way a piece of software can still open up a socket (with \u0026ldquo;root permissions\u0026rdquo;), while not being able to load a new kernel driver. For more details about capabilities, see our previous blog post Linux Capabilities 101.\nContainers will run with a limited capability set. So even if someone breaks into the container, the host system is to some extent protected.\nExamples:\nMounting operations Access to raw sockets (prevent opening privileged ports, spoofing) Some file system operations (mkdev, chown, chattrs) Loading kernel modules The configuration and usage of capabilities will be covered later. For now it is good to know that Docker by default drops a list of capabilities:\nCAP_AUDIT_WRITE = Audit log write access CAP_AUDIT_CONTROL = Configure Linux Audit subsystem CAP_MAC_OVERRIDE = Override kernel MAC policy CAP_MAC_ADMIN = Configure kernel MAC policy CAP_NET_ADMIN = Configure networking CAP_SETPCAP = Process capabilities CAP_SYS_MODULE = Insert and remove kernel modules CAP_SYS_NICE = Priority of processes CAP_SYS_PACCT = Process accounting CAP_SYS_RAWIO = Modify kernel memory CAP_SYS_RESOURCE = Resource Limits CAP_SYS_TIME = System clock alteration CAP_SYS_TTY_CONFIG = Configure tty devices CAP_SYSLOG = Kernel syslogging (printk) CAP_SYS_ADMIN = All others Usage of seccomp Secure Computing, or seccomp, helps with the creation of sandboxes. It does so by defining what system calls should be blocked. The latest version of seccomp provides this syscall filtering by using the Berkeley Packet Filter (BPF), previously used for filtering network traffic.\nContainers currently have the following syscalls disabled (since LXC 1.0.5):\nkexec_load(2) open_by_handle_at(2) init_module(2) finit_module(2) delete_module(2) When any of the blocked syscalls is made, the kernel will send a SIGKILL signal to stop the related process.\nDigital Signature Verification Starting with Docker version 1.3.0 all images are verified after downloading. This is a great step in enhancing the level of trust for downloads. This is why we have done this for our auditing tool Lynis as well. Like our website, Docker is also providing their website HTTPS-only. Another level of trust and ensuring you are are the right place, downloading the tools from Docker.\nCurrent issues with Docker Root permissions Right now there is a small issue left with Docker, which is the requirement of running the daemon with root permissions. Docker is aware of it and plans to define well-audited sub-processes, which do not longer require root permissions. Additionally, each sub-process will run with a very limited scope, increasing the security level of each component and enhancing stability.\nLack of full User namespace implementation Currently there is still no full user namespace implementation. Something which is out of the control of Docker. When the LXC userland tools are evolved and include the support, Docker can leverage the possibilities. The first actions like user mapping is available, so full support is expected soon.\nUser 0 in container = User 0 on host One of the risks due to the missing User namespaces, is that the mapping of users from the host to the containers is still an one-to-one mapping. Previously user 0 in the container was equal to 0 on the host. In other words, if you container is compromised, it doesn\u0026rsquo;t take much to compromise the full host. Fortunately this is work in progress. LXC already supports a mapping option, to map user ID 0 in the container to another (high) ID on the host.\nDefault allow all By default all IP traffic is allowed between containers. This means they can ping each other, but also send other forms of traffic. It would have been better if Docker applied a \u0026ldquo;deny all by default\u0026rdquo; principle. This forces the maintainer of the container to think about what kind of traffic is needed between individual containers.\nFortunately traffic can be filtered and is absolute advised for systems in production.\nBest Practices With all these risks and possibilities, lets extract some of the lessons. These best practices help you create more safe services and enhance the security of existing containers.\nDo not run software as root This tip might sound to simple, but still many developers run their software as the root user. Containers still can not contain properly, which might result in a full host compromise if a container is compromised. Therefore, run your software packages like you would run them on a normal host.\nUse Docker in combination with AppArmor/SELinux/TOMOYO Ubuntu comes with ready-to-use AppArmor templates for LXC. It is always a good thing to know what your software does. This includes knowing what paths and permissions your software components need to function properly. With this information each piece can be restricted to the bare minimum needed, preventing permission escalation and unauthorized information disclosure (or worse).\nTo achieve the right policies, make sure to monitoring your applications from the start, including the related framework you are using. Each of them provides the means to monitor, so use them.\nWithin the container configuration the related AppArmor profile can be defined with lxc.aa_profile.\nUse seccomp to limit syscalls Support for seccomp is available to (at least) CentOS, Debian, Fedora, Gentoo, Oracle, Plamo and Ubuntu. You can use seccomp by altering the container configuration and define the seccomp rule set to be used:\nlxc.seccomp = /usr/share/lxc/config/common.seccomp\nFor Docker this functionality can be activated by using the -lxc-conf parameter to docker run.\nLXC configuration option: lxc.seccomp\nLimit traffic with iptables By default all containers use the docker0 interface as a bridge. Like on a normal host you can limit traffic, to block unauthorized traffic streams.\nFor full details we suggest to read the Advanced networking at Docker.\nGRSEC and PaX When possible, use a hardened Linux kernel, with kernel patches. Grsecurity and PaX are two examples which help in hardening the host system.\nUsing user mappings To counter the issue that user 0 in a particular container equals root on the host system, LXC allows you to remap user and group IDs. The configuration file entry would look something like this:\nlxc.id_map = u 0 100000 65536\nlxc.id_map = g 0 100000 65536\nThis maps the first 65536 user and group IDs within the container to 100000-165536 on the host. The related files on the host are /etc/subuid and /etc/subgid. This mapping technique is named subordinate IDs, hence the \u0026ldquo;sub\u0026rdquo; prefixes.\nFor Docker this means adding it as a -lxc-conf parameter to docker run:\ndocker run -lxc-conf=\u0026quot;lxc.id_map = u 0 100000 65536\u0026quot; -lxc-conf=\u0026quot;lxc.id_map = g 0 100000 65536\u0026quot;\nUse -cap-drop and -cap-add (since Docker 1.2.0)\nWith the earlier described Linux capabilities we can tell Docker what specific roles should be given to a container. Actually, we can use both the add and drop option together. By first allowing all, then dropping some capabilities, we can limit the capabilities. The short version is just dropping permissions. The alternative is just adding the related capabilities, which more resembles the \u0026ldquo;deny all\u0026rdquo; principle.\ndocker run -cap-drop=CHOWN\ndocker run -cap-add=ALL -cap-drop=MKNOD\nA (fairly) safe list of capabilities to drop are:\naudit_control audit_write mac_admin mac_override mknod setfcap setpcap sys_admin sys_boot sys_module sys_nice sys_pacct sys_rawio sys_resource sys_time sys_tty_config See the capabilities(7) man page for all details about these capabilities.\nLXC configuration options: lxc.cap.drop and lxc.cap.keep\nDo not run SSH in containers Use \u0026ldquo;docker exec -it mycontainer bash\u0026rdquo; instead to manage your containers.\nDo not run -privileged on containers (since Docker 1.3.0)\nFor containers which already have SELinux/AppArmor support, use -security-opt instead. This gives it the appropriate security profile, instead of giving away too much permissions within the container.\ndocker run -security-opt label:type:svirt_apache -i -t centos \\ bash\nRelated options for SELinux:\n-security-opt=\u0026ldquo;label:user:USER\u0026rdquo; (set label user) -security-opt=\u0026ldquo;label:role:ROLE\u0026rdquo; (set label role) -security-opt=\u0026ldquo;label:type:TYPE\u0026rdquo; (set label type) -security-opt=\u0026ldquo;label:level:LEVEL\u0026rdquo; (set label level) -security-opt=\u0026ldquo;label:disable\u0026rdquo; (disable label confinement completely) Options for AppArmor:\n-secutity-opt=\u0026ldquo;apparmor:PROFILE\u0026rdquo; (set AppArmor profile) For more options have a look at the Docker Run documentation.\nUpgrade your Docker version on a regular basis Most software packages have bugs, small programming errors. With Docker also being under heavy development, bugs are solved and new features added. Regular updating and making Docker part of your software patch management process, is advised.\nSecure Docker client connections Set the DOCKER_HOST and DOCKER_TLS_VERIFY variable, to use TLS for connecting to Docker instances. See Docker HTTPS for detailed instructions on setting this up.\nDocker Tooling Security tools Dagda Vulnerabilities in Docker Over the years vulnerabilities will show up. Some of the Docker security messages are collected for archival purposes:\nDocker 1.3.3 fixes Docker 1.3.2 fixes Docker 1.3.1 fixes Citing Sources Used references and sources, include the following websites:\nBlog of Docker (original article no longer available) Security Risks and Benefits of Docker Application Containers - Blog of Lenny Zeltser News about Linux containers (LXC) - Useful changelogs for LXC Docker Run Reference - Docker Run Reference Found something outdated in this article? Add it to the comments. This article is kept up-to-date on a regular basis, together with the developments of Docker itself.\n","permalink":"https://linux-audit.com/docker-security-best-practices-for-your-vessel-and-containers/","tags":["capabilities","containers","docker","linux","security"],"title":"Docker Security: Best Practices for your Vessel and Containers"},{"categories":["Auditing","Intrusion Detection","Linux"],"contents":"Linux has several solutions to monitor what happens with your data. From changing contents to who accessed particular information, and at what time.\nFor our auditing toolkit Lynis, we researched and tested several solutions over the last few years. In this article we have a look at these solutions to monitor file access, changes and modifications to the data and beyond.\nWhat is Data? Data is a collection of bits, ordered in such a way it gives meaning to humans. The related information stored in data blocks, can be as simple as text, or become a visible representation like an image. Data is usually the most important part on a system, which means it has to be properly safeguarded.\nData versus Meta-data Besides the information stored for us, the system needs to store a little bit of information as well. For example a data block on disk, might need some supporting information to know where it is stored. This data is usually not useful for us, but certainly for the system to retrieve the information, especially when we ask for it. This \u0026ldquo;data about data\u0026rdquo; is called meta-data. So besides protecting data, we have to take the protection of meta-data in mind as well.\nMonitoring File Access The first level of monitoring is who is accessing specific files. This helps us understand what particular files are being accessed, by what process and by whom. To accomplish this task, we can use the Linux Audit Framework. The framework is written by Red Hat and uses \u0026ldquo;watches\u0026rdquo; on files and directories to determine what should be monitored. Additionally it can monitor processes, including the underlying system calls which are performed by them.\nAdding watches To protect our kernel configuration, we can determine who accesses the /etc/sysctl.conf file. This file stores kernel settings, so it interesting to start with this file. To have this file monitored, we use auditctl and add a watch on the file.\nauditctl -w /etc/sysctl.conf -p a -k kernel\nThe parameter -w sets the watch, followed by the file name. The -p defines the related permission action (a = attribute change, r = read, w = write, x = execute). It looks similar to file permissions, but actually it is slightly different. With the -k we define a custom key, which simplifies searching at a later moment. It is also helpful to categorize events.\nReporting watches Now we have defined our watch, we can search for it with the earlier defined key using the ausearch command.\nausearch -k kernel\nWhen looking at this output, you might be overwhelmed by all the fields available. Additionally some fields actually have rather strange values, like an architecture of c000003e (which actually equals x86_64).\nThe most important fields are the purple box, showing what object was hit and the green box revealing the process (or binary), followed by the defined key. In this case both the cat command and vim editor have opened the file\nIn this screenshot we can also see a failed syscall in the yellow box, with the value 89. To determine what syscall this is, we first have to look it up:\nausyscall -dump\nThis will show all available syscalls for our particular system architecture. So in this case a call to \u0026ldquo;getrusage\u0026rdquo;, to retrieve process statistics from the kernel.\nMonitoring specific functions We can use the Linux audit framework also for monitoring specific system calls, or functions. We have to use the -S followed by the system call.\nauditctl -a always,exit -S openat -F success=1\nThe -a always,exit defines to write out an event at exit time of the related system call.\nFor example when you want to monitor all successful \u0026ldquo;openat\u0026rdquo; calls, add this system call and tell auditctl only to log successful requests. In this case you might get a message that the system call is unclear, as it is found on multiple architectures. Find the related system call ID with ausyscall openat and add the ID instead. Even better is specifying the architecture together with the system call, as it is easier to read (example: -F arch=b64 -S openat).\nFor more tips regarding the Linux audit framework, have a look at our other article Configuring \u0026amp; Auditing Linux Systems with the Audit Daemon.\nFile Integrity Monitoring Another interesting level to monitor file changes, is by implementing file integrity tooling. Linux has several options for this, varying from simple tools up to kernel modules.\nFile Integrity Tools The easiest way to verify if a file has been changed, is using tools. Simple tools like md5sum or shasum can help with detecting changes. Also specialized tools like AIDE and Samhain are a great help to set-up automatic monitoring and alerting.\nSince setting up these tools are worth a blog post of their own, it will be covered in a separate post.\nIntegrity Measurement Architecture (IMA) The most extensive option is monitoring files with IMA . This security module allows the system to create and monitor hashes for files and block unauthorized changes.\nIMA has a few modes it can operate in, like fix and appraise. In \u0026ldquo;fix mode\u0026rdquo; the system allows the administrator to set hash values along each file. These hashes are small strings of text to help the system detect changes and are stored in extended attributes (xattrs) of the file system.\nDigital signatures\nAdditionally IMA supports digital signing. This ensures you that the contents of the file is correct (or unaltered). Additionally because it is signed, you can validate the signature. So if a file is to be changed, it also needs proper signing.\nSince IMA is a very extensive way of monitoring, we will cover more in other blogs posts. It\u0026rsquo;s a very exciting subject and a great help to protect your data.\nExtended Verification Method (EVM) Where IMA monitors the file contents, EVM performs monitoring of the file attributes. It also allows hashing and digital signing. It\u0026rsquo;s a great extension to IMA, to ensure that both contents and the attributes of a file are being unaltered.\nMonitoring File Attributes To monitor file permissions, we can also use the audit framework. File permissions and ownership are part of the file attributes. The file attributes can be monitored with \u0026ldquo;-p a\u0026rdquo;.\nAdditionally, we can use the earlier covered EVM to ensure attributes are not changed by an unauthorized process or person.\nConclusion Now we have looked at some of the tools, it should be clear that a lot of areas can be monitored on Linux systems. It is up to the administrator to define what files should be monitored and to which extent. From simply logging changes to attributes with the Linux audit framework, up to fully blocking altered files with IMA and EVM.\n","permalink":"https://linux-audit.com/monitoring-linux-file-access-changes-and-modifications/","tags":["auditctl","ausearch","file integrity","file integrity monitoring","system integrity"],"title":"Monitoring Linux File access, Changes and Data Modifications"},{"categories":["Software","Vulnerabilities"],"contents":"If you audit systems on a regular basis, you eventually will come across an OpenBSD system. OpenBSD is known for its heavy focus on security, resulting in an operating system with a low footprint and well-audited source code.\nWhile most operating systems are pretty secure, they quickly will introduce new security holes when installing external software components. Although OpenBSD does careful checks for packages they add, those might be containing still a vulnerability, waiting to be discovered. So in this article we have a close look at dealing with packages and what to look for when auditing them.\nOpenBSD Software Packages Packages versus Ports OpenBSD has the ability to install a package or a port. The latter is nothing more than a smart reference to the original source file, completed with operating system specific patches. This enables the system to actually successfully build the program.\nPackages on the other hand, are precompiled binaries, with configuration files and installation instructions. In this case the package manager knows exactly where to place each file and with what permissions.\nOpenBSD advices users to use packages, as they are optimized and carefully examined. Besides that, it will limit the hassle users might have getting ports compiled, due to changes in the operational environment. The smallest change in a default compiler option might influence its behavior, which are better tailored for the dedicated people who build the packages.\nVulnerability Scanning OpenBSD does not maintain a specific vulnerability database, like for example NetBSD does. However to determine what (security) updates are available, it is wise to follow the related mailing lists, and simply stay up-to-date with software packages.\nChecking for patches can be done via the pkg_add utility. Use the update mode (-u) and don\u0026rsquo;t allow it to actually do the update (-s). The -x and -I are for non-interactive display and mode.\npkg_add -Isxu\nNote: this will only work for snapshots, and upgrading between releases.\nDigital Signatures Starting with OpenBSD 5.5 the tool signify was introduced. The name is a combination of \u0026ldquo;sign\u0026rdquo; and \u0026ldquo;verify\u0026rdquo;. So it is no surprise that its goal is to sign and verify OpenBSD releases, includes underlying components like packages.\nSignify components To allow OpenBSD software packages being signed and verified, they are split into two phases. In the first phase a package maintainer builds the package. Then the package is signed with a secret key (private key). When a user wants to install the related package, the linked public key will then be used to verify the digital signature. If it matches, installation continues. If not, the installation is aborted.\nPublic keys\nOpenBSD distributes the keys packaged with the base system itself.\n# ls -l /etc/signify/* -rw-r-r- 1 root wheel 104 Aug 8 08:05 /etc/signify/openbsd-55-base.pub -rw-r-r- 1 root wheel 108 Aug 8 08:05 /etc/signify/openbsd-55-fw.pub -rw-r-r- 1 root wheel 108 Aug 8 08:05 /etc/signify/openbsd-55-pkg.pub -rw-r-r- 1 root wheel 104 Aug 8 08:05 /etc/signify/openbsd-56-base.pub -rw-r-r- 1 root wheel 108 Aug 8 08:05 /etc/signify/openbsd-56-fw.pub -rw-r-r- 1 root wheel 108 Aug 8 08:05 /etc/signify/openbsd-56-pkg.pub -rw-r-r- 1 root wheel 104 Aug 8 08:05 /etc/signify/openbsd-57-base.pub -rw-r-r- 1 root wheel 108 Aug 8 08:05 /etc/signify/openbsd-57-fw.pub -rw-r-r- 1 root wheel 108 Aug 8 08:05 /etc/signify/openbsd-57-pkg.pub These files represent the base system, firmware, and packages.\nChecks in pkg_*\nTo properly use digital signatures, they should be used before actually installing new components on the system. So the utility pkg_add got extended to do these checks. Next in line is pkg_info, which got a new -C option to show the digital certificate information as well. The result of using this option is a small line stating \u0026ldquo;reportedly signed by\u0026rdquo; followed by the name of the public key.\nDigitally signed software in OpenBSD pkg_info\nSignify tool\nAnother option is using the signify tool directly. For example when downloading new files, use the related key and signed hash file to verify the file integrity.\n# ftp ftp://ftp.eu.openbsd.org/pub/OpenBSD/5.6/amd64/{bsd,SHA256.sig} # signify -C -p /etc/signify/openbsd-56-base.pub -x SHA256.sig bsd Signature Verified bsd: OK Conclusion OpenBSD is known for its security and it was actually a surprise that signed software was only introduced in OpenBSD 5.5. But now we have this available, the chain of package release, up to installation can be checked more easily. This is definitely another good step to keep systems secure!\n","permalink":"https://linux-audit.com/vulnerabilities/vulnerabilities-and-digital-signatures-for-openbsd-software-packages/","tags":["software","software management","vulnerabilities"],"title":"Vulnerabilities and Digital Signatures for OpenBSD Software Packages"},{"categories":["Compliance","PCI DSS compliance"],"contents":"PCI describes in control 10.2.4 to monitor for \u0026ldquo;invalid logical access attempts\u0026rdquo;. Another way of saying to monitor attempts which are not allowed, like accessing a file you are not supposed to. Another indication might be brute force attempts to log in, which result in several failed logins.\nTo monitor for invalid access attempts, we can use the Linux Auditing Framework. This framework has been created and maintained by Red Hat over the years. It is a great tool for auditing and to help with PCI DSS compliance on Linux based systems.\nMonitoring access attempts To filter out invalid access attempts, we can monitor all system calls which return an \u0026ldquo;access denied\u0026rdquo; error. We can translate this into the following audit rules:\n-a always,exit -F arch=b32 -S all -F exit=-13 -k access-denied\n-a always,exit -F arch=b64 -S all -F exit=-13 -k access-denied\n-S all = All system calls.\n-F exit=-13 = Exit code of -13, , which equals access denied.\nThese rules can be added to the /etc/audit/audit.rules file or directly via auditctl.\nBy defining a key, we can quickly find it later with ausearch.\nTesting the rule Now try to access a file as a non-privileged user (e.g. cat /etc/shadow). It should log the access attempt by creating a new event in the audit log.\nBy using the ausearch utility, we can search events in this category.\nausearch -k access-denied\nThe Linux audit framework provides great ways to monitor files, directories and processes. With the right filters, it is a great addition for companies who would like to become PCI compliant.\nChecking for failed logins By default, the audit framework can also gather failed logins using the ausearch utility.\n# ausearch -message USER_LOGIN -success no -interpret type=USER_LOGIN msg=audit(12/05/2014 10:16:25.133:372) : pid=2594 uid=root auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg=\u0026lsquo;op=login acct=(unknown user) exe=/usr/sbin/sshd hostname=? addr=192.168.1.2 terminal=ssh res=failed\u0026rsquo;\nMost of the rules we share on this blog are also being used for automated testing in our auditing tool Lynis and related compliance plugins. If you want to automate your PCI DSS security audits, start with the free Lynis tool .\nHappy auditing!\n","permalink":"https://linux-audit.com/pci-dss-v3-linux-invalid-logical-access-attempts-10-2-4/","tags":["compliance","linux","pci dss"],"title":"PCI DSS (v3) Linux: Invalid logical access attempts (10.2.4)"},{"categories":["Malware"],"contents":"Rootkits are considered to be one of the most tricky pieces of malware. Usually they are loaded onto the system by exploiting weaknesses in software. Next phase is being installed and hide as good as possible, to prevent detection. We have a look at a few security measures you can take to prevent this kind of threat.\nSystem Protection Kernel The kernel is the brain of the software system and decides what should be executed by the central processing unit. Any \u0026ldquo;damage\u0026rdquo; to this system decreases the integrity of the system and your data. So protecting the kernel against unauthorized modifications is an important step in keeping system secure.\nModules Linux was originally a monolithic kernel, consisting of all functions in one piece of software. Nowadays its flexible, allowing additional functionality to be loaded when required. The downside of this is the chance that malicious people use this to load new kernel module, which in their turn alter how the kernel functions. For example when requesting a process listing, the kernel might leave out specific processes. Rootkits use these kind of tricks to prevent detection and it shows (pun intended).\nBinaries Next in line are software components and system utilities in particular. Command like ps, ls, netstat, and others are targeted by rootkits, to avoid detection.\nIntrusion Detection While prevention is a good thing, detection is usually even more valuable. After all, if you can\u0026rsquo;t detect an intrusion, how do you know for sure you are safe? Let\u0026rsquo;s have a look at a few possibilities to perform malware and intrusion detection.\nFile Integrity Monitoring Objective: Monitor changes to files and binaries in particular.\nFile integrity monitoring can be implemented on different levels. Even a small script collecting all hashes could be a start. Better would be the usage of file integrity tools, which can detect changes and perform intrusion detection. Common examples include AIDE, AFICK and Samhain.\nIf you want to go even a step further, you can use IMA and EVM, which are Linux security modules to do file integrity for all files. When the kernel sees a file which is changed, it refuses to execute the file. This is definitely the maximum level of protection.\nMalware Detection Tools Object: Use proper tooling to detect malware.\nTools like Rootkit Hunter help with detection some forms of rootkits. ClamAV and LMD are other great examples of tools who can search for traces of malware.\nIntrusion Prevention While full system hardening is preferred, we focus on some powerful methods to prevent the biggest threats to break in.\nUpgrade the System Objective: stay up-to-date with software components\nUpgrade on a regular basis software components. Also implement a strategy on how to deal with security upgrades in particular. Most Linux distributions have a way to split these security patches, so you can easily focus on those first.\nAdditionally it is wise to check for a required reboot. Patching is a great first step, but the kernel itself should be reloaded if there is a new version. Sure, we also know uptime is important and your customers don\u0026rsquo;t like reboots. But there is personal data at stake and usually more important than that tiny window of downtime.\nDisable kernel modules Objective: Disable loading of kernel modules, if system does not require it.\nCheck lsmod output, blacklist modules which are not being used by using the blacklist file. Additionally, disable loading all kernel modules for full protection.\nBlacklist Kernel Modules Create a custom file in /etc/modprobe.d directory, like custom-blacklisted.conf.\nFor example our output looks like this:\n# lsmod Module Size Used by ppdev 17671 0 serio_raw 13462 0 i2c_piix4 22155 0 joydev 17381 0 parport_pc 32701 0 mac_hid 13205 0 lp 17759 0 parport 42348 3 lp,ppdev,parport_pc xts 12914 1 gf128mul 14951 1 xts dm_crypt 23177 1 hid_generic 12548 0 usbhid 52570 0 hid 106148 2 hid_generic,usbhid psmouse 106678 0 ahci 25819 2 libahci 32560 1 ahci e1000 145174 0 So let\u0026rsquo;s blacklist these items with:\nblacklist e1000\nblacklist psmouse\nblacklist usbhid\nblacklist hid_generic\nblacklist lp\nblacklist mac_hid\nblacklist ppdev\nblacklist serio_raw\nblacklist i2c_piix4\nblacklist joydev\nblacklist parport_pc\nSurprisingly we still see some show up after rebooting the system.\n# lsmod Module Size Used by lp 17759 0 parport 42348 1 lp xts 12914 1 gf128mul 14951 1 xts dm_crypt 23177 1 hid_generic 12548 0 usbhid 52570 0 hid 106148 2 hid_generic,usbhid psmouse 106678 0 ahci 25819 2 libahci 32560 1 ahci e1000 145174 0 Some modules are still being loaded, like lp, usbhid and e100. This is due the modules being loaded at boot time. Some are actually needed, although the output looks like the module is being unused. For example e1000 is our network driver.\nThe lp module on the other hand is not needed, as can be seen by the dmesg output for lp: [ 219.078098] lp: driver loaded but no devices found.\nSo we should disable this driver from /etc/modules:\nKernel modules to be loaded at boot time\nAfter adding a hash to the line starting with \u0026ldquo;lp\u0026rdquo;, we reboot the system another time. This time it is gone. Due to the blacklist line, services like udev can not load it anymore. However, modprobe still can. So we have to put in another layer of defense: disable loading new kernel modules.\nDisabling Loading of Kernel Modules To disable new modules from being loaded, a kernel setting have to be put in place via the sysctl command. By setting the value for kernel.modules_disabled to 1, no more kernel modules can be loaded.\nLinux kernel modules being disabled with sysctl key\nAs can be seen in the screenshot above, the system refuses to turn on the setting again (even though it displays the value of 0). The great thing of this option is that attackers can\u0026rsquo;t disable this setting without rebooting the system. To keep things secure, you might want to monitor the system for unauthorized reboots as well.\n","permalink":"https://linux-audit.com/malware/monitoring-linux-systems-for-rootkits/","tags":["binaries","lsmod","malware","monitoring","rootkit"],"title":"Monitoring Linux Systems for Rootkits"},{"categories":["Compliance","PCI DSS compliance"],"contents":"Companies who need to comply with the PCI DSS standard need to log all actions which are executed by the root user or those accounts with similar administrative privileges.\n10.2.2 Verify all actions taken by any individual with root or administrative privileges are logged.\nThe Linux kernel allows the monitoring of executed commands. This monitoring and logging can be done with the Linux audit framework. Using this framework, we can monitor the right system calls and create an audit trail. It is also called Linux accounting. Such accounting is similar to the call history on your mobile phone bill.\nConfigure logging To capture executed commands, we can monitor the execve(2) system call. Use auditctl to add a rule, or by defining in /etc/audit/audit.rules .\nauditctl -a exit,always -F arch=b64 -S execve -k root-commands\nauditctl -a exit,always -F arch=b32 -S execve -k root-commands\nNote: this captures the 32-bit and 64-bit requests.\nConfirm the rules are loaded with the auditctl command.\nauditctl -l\nThe output will be looking something like this:\nLIST_RULES: exit,always arch=3221225534 (0xc000003e) key=root-commands syscall=execve\nLIST_RULES: exit,always arch=1073741827 (0x40000003) key=root-commands syscall=execve\nIf this works, we can improve the audit rule, by limiting it only the root user. This is done by adding the euid or effective user ID.\nauditctl -a exit,always -F arch=b64 -F euid=0 -S execve -k root-commands\nauditctl -a exit,always -F arch=b32 -F euid=0 -S execve -k root-commands\nAnother alternative is to filter by the execve system call, is using a permissions filter. In this option, we look at all calls, but only log those that perform write, change to attributes or execute an action. We still will restrict this only to what the root user or its equivalent.\nauditctl -a exit,always -S all -F euid=0 -F perm=awx -k root-commands\nIt\u0026rsquo;s up to you what you prefer. We suggest testing in your environment to decide what gives a proper amount of accounting without overloading your system.\nNote: use the euid filter, as auid will not account for sudo related commands.\nTesting Now we have defined the rules, it is time for testing them. To emulate this, we run the echo command.\nRunning echo command with sudo:\ntime-\u0026gt;Wed Dec 24 02:56:21 2014 type=PATH msg=audit(1419386181.134:340876): item=1 name=(null) inode=1967930 dev=08:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PATH msg=audit(1419386181.134:340876): item=0 name=\u0026#34;**/usr/bin/sudo**\u0026#34; inode=149160 dev=08:02 mode=0104755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=CWD msg=audit(1419386181.134:340876): cwd=\u0026#34;/home/michael\u0026#34; type=EXECVE msg=audit(1419386181.134:340876): argc=3 a0=\u0026#34;**sudo**\u0026#34; a1=\u0026#34;**echo**\u0026#34; a2=\u0026#34;**test**\u0026#34; type=BPRM\\_FCAPS msg=audit(1419386181.134:340876): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old\\_pp=0000000000000000 old\\_pi=0000000000000000 old\\_pe=0000000000000000 new\\_pp=ffffffffffffffff new\\_pi=0000000000000000 new_pe=ffffffffffffffff type=SYSCALL msg=audit(1419386181.134:340876): arch=c000003e syscall=59 success=yes exit=0 a0=1082568 a1=ec8a08 a2=10dd008 a3=7fffb8fa1e50 items=2 ppid=15400 pid=15535 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts5 ses=221 comm=\u0026#34;sudo\u0026#34; exe=\u0026#34;**/usr/bin/sudo**\u0026#34; key=\u0026#34;root-commands\u0026#34; Running the same command as root, by evoking the /bin/echo command:\ntime-\u0026gt;Wed Dec 24 02:57:41 2014 type=PATH msg=audit(1419386261.026:340974): item=1 name=(null) inode=1967930 dev=08:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=PATH msg=audit(1419386261.026:340974): item=0 name=\u0026#34;**/bin/echo**\u0026#34; inode=135948 dev=08:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL type=CWD msg=audit(1419386261.026:340974): cwd=\u0026#34;/root\u0026#34; type=EXECVE msg=audit(1419386261.026:340974): argc=2 a0=\u0026#34;**/bin/echo**\u0026#34; a1=\u0026#34;**test**\u0026#34; type=SYSCALL msg=audit(1419386261.026:340974): arch=c000003e syscall=59 success=yes exit=0 a0=18f1648 a1=18f2a48 a2=1af8008 a3=7fff98be9820 items=2 ppid=15610 pid=15632 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts7 ses=223 comm=\u0026#34;echo\u0026#34; exe=\u0026#34;**/bin/echo**\u0026#34; key=\u0026#34;root-commands\u0026#34; Note: Keep in mind that not all commands are logged. All built-in shell functions will NOT use the execve system call, therefore they are not logged.\nThis guide is supporting documentation for our Lynis Enterprise solution. It helps companies getting compliant with PCI DSS . We help to automate the hardening and auditing process, so you don\u0026rsquo;t have to check everything manually.\n","permalink":"https://linux-audit.com/compliance/pci-dss-logging-of-administrative-actions-with-root-privileges/","tags":["accounting","auditing","compliance","linux","pci dss","sudo"],"title":"Logging of administrative actions with root privileges"},{"categories":null,"contents":"","permalink":"https://linux-audit.com/search/","tags":["website"],"title":"Search results"},{"categories":["Lynis"],"contents":"Professionals ask us often how Lynis is different than Tenable Nessus . As the original author of Lynis, let me address that very interesting question.\nDifferent goal Nessus is focused on vulnerability scanning, or in other words, finding weaknesses in you environment. The huge amount of plugins and their actions show that this is the primary focus. Along the way it started to implement others services, like compliance checking.\nLynis also detects vulnerabilities, but that is not its main goal. Primary focus for Lynis is auditing the system and helping the user with follow-up: system hardening. Lynis is hungry for data, so it can combine things and give the user better advice.\nFocus on Detection versus Prevention Both tools focus on detection. Where Nessus definitely discovers a lot of weaknesses, Lynis checks for more than just that. It goes beyond just checking for a version number or configuration file, by also confirming that the configuration is working. Lets say you configured a few DNS servers, but some are not reachable. Lynis will discover this and tell you about it. Sometimes weaknesses are not software bugs, but simply configuration errors.\nAnother area in which Lynis goes further, is in the area of of prevention. It will actually encourage users to improve their security defenses by providing a hardening index, suggestions and follow-up steps. Sure, Nessus has reports, but do those really encourage hardening when they are also filled with a lot of informational data?\nLogging Noise Performing scans via the network will definitely give a lot of noise in your log files. Users of network based scanners know this and these scans are easy recognizable in your log files. This noise is caused by the \u0026ldquo;active\u0026rdquo; component of network scanners. It first has to complete the enumeration phase, to discover systems and services. They need to know what is running on a particular device, before starting with the next phase of exploiting them.\nIn the case of Lynis, a host based scanner, log files will remain calm. The tool will look directly in configuration files, check process listing and query version numbers locally. This means no guessing and providing factual details from the source itself. It also saves a lot of time and more information on the system can be detected than outside.\nNessus has an option to do SSH based logins, an addition to its active scans. It definitely helps detecting more details of the systems. Still, it does meet the detailed insights which an auditing tool provides.\nSpecialization Regarding the platform there is a interesting difference. Nessus supports all operating systems and can even scan unknown devices. Great option for scanning whole networks.\nLynis is very specialized in Unix based systems, therefore only supports Linux, Unix and Mac OS. If you want to do a vulnerability scan of your full network, you may want to use Nessus. When auditing and hardening is your goal, a specialized tool is more precise and better for your follow-up. In that case Lynis is the way to go.\nConclusion Both Nessus and Lynis bring interesting things to the table. Do we like Nessus? Sure! Do we like Lynis even more? Yes, we do. In the end it is about personal preferences, but more importantly, about the goal: Want to do only vulnerability scanner, then we can advise a tool like OpenVAS or Nessus. If auditing and hardening is the goal, Lynis will definitely win in that area.\n","permalink":"https://linux-audit.com/product-comparison-lynis-vs-nessus/","tags":["comparison","lynis"],"title":"Product comparison: Lynis VS Nessus"},{"categories":["System Administration","Time"],"contents":"The Secure Alternative for ntpd, ntpdate and rdate\nThe common protocol to synchronize the time, is named Network Time Protocol, or NTP. While this protocol works great for synchronizing systems to one or more multiple time sources, it is not always easy to set-up. One alternative is using tlsdate, a secure replacement to keep your systems in sync.\nAbout the Project The software is written in 2012 by Jacob Appelbaum and can be found at GitHub: tlsdate . With the primary goal of securely synchronizing time, it uses an an unintentional part in the TLS protocol. For Chromium OS , tlsdate is actual the default synchronization tool.\nTLS and tlsdate Within the specification of the TLS protocol (RFC 2246 3 ), there are two functions (ServerHello and ClientHello), which should have a timestamp embedded in their responses. This is the time of the server, and can be used to compare it with the local time. The interesting part is that is can also be used for synchronizing the time. While not having the same accuracy as NTP, it is fairly precise.\nUsing tlsdate After installing tlsdate on your distribution, start tlsdate with the -h option to get all options.\n# tlsdate -h tlsdate usage: [-h|-help] [-s|-skip-verification] [-n|-dont-set-clock] [-H|-host] [hostname|ip] [-p|-port] [port number] [-P|-protocol] [sslv23|sslv3|tlsv1] [-C|-certcontainer] [dirname|filename] [-v|-verbose] [-V|-showtime] [human|raw] [-t|-timewarp] [-l|-leap] [-x|-proxy] [url] [-w|-http] To query the time, we can set the hostname with -H, provide the -v for verbose output and tell tlsdate not to sync the time yet (-n).\nTime synchronization with tlsdate\nIn this example we used one of the mail servers of Google to fetch the time, using TLSv1.\nEven if you don\u0026rsquo;t want to use tlsdate to synchronize your systems, it can be used to query the time on a different system.\nPros The biggest pros of tlsdate are:\nEasy of use The tlsdate toolkit is easy to use, with clear configuration files and option.\nSecurity By leveraging TLS, synchronization is secure.\nSpeed of synchronizing The tool is quick and directly synchronizes the time.\nCons On the side of the cons we have:\nPrecision of the time synchronization process For most systems however, having a precision within a few seconds is acceptable. But in some cases you want to know exactly what event happened when, especially for forensics and comparing events on multiple systems. Even a few milliseconds can be too much to accurately describe what happened.\nTLS 1.3 and tlsdate In the upcoming version of TLS, tlsdate might not function anymore.\nQuality of Time At this moment tlsdate lacks features like how \u0026ldquo;drifting\u0026rdquo; of time is managed, the amount of time difference due to slight differences in chip sets and electronic components. Another one is the lack of compensation for network latency. Last, but not least, the quality of the time source itself can not be measured. Where NTP uses a process similar to voting, tlsdate simply uses one source.\nConclusion For desktops and most servers, tlsdate might be a great alternative to the existing arsenal of time synchronization services. If you want precision, we suggest to have a look at OpenNTPD , from the folks behind OpenBSD. If something is secure and well-audited, it\u0026rsquo;s the code coming from these projects.\n","permalink":"https://linux-audit.com/tlsdate-the-secure-alternative-for-ntpd-ntpdate-and-rdate/","tags":["ntp","time"],"title":"tlsdate: The Secure Alternative for ntpd, ntpdate and rdate"},{"categories":["Auditing","Compliance","Hardening"],"contents":"An alternative to CIS Benchmarks and hardening guides\nHardening guides, and the CIS benchmarks in particular, are a great resource to check your system for possible weaknesses and conduct system hardening. But who has the time to read it cover to cover, and apply every single step? In this article, we have a look at the alternative: open source auditing tools.\nTime.. Hardening is a time-consuming task. As security specialists, we know that. It involves many small steps, followed by even more testing and troubleshooting. Unfortunately, time is something we can spend only once, making it a scarce resource.\nTo save time on reading extensive hardening guides, we suggest our users to use proper auditing tooling instead. It helps in automating repeating tasks, saving precious time. This time is better spent on the hardening itself, right? Secondly, using tooling we can quicker detect newly introduced security weaknesses. For example, caused by an unaware developer or colleague. After all, you want to safeguard your earlier work, avoiding someone performing an \u0026ldquo;undo\u0026rdquo; on it.\nCIS Benchmarks Back to the friendly people from CIS. They create extensive hardening guides, named CIS benchmarks. These benchmarks are available for most common platforms available, like Windows, several Linux distributions, Solaris, and others.\nCIS have their own staff and get additional help from seasoned professionals. These professionals are people usually working for multinationals and consultancy firms and commonly named subject matter experts (SME) in their specific field of expertise.\nCIS uses \u0026ldquo;consensus teams\u0026rdquo;. In other words, teams with security professionals who have discussions to decide what kind of advice is suitable for most environments. This clearly improves the quality of the guides, but can also be a flaw. For example, if you have different demands for your environment, you still have to consider each item in the guide. Sometimes a control might be too strict, and sometimes it simply is not enough to protect your precious resources. But in the end, we believe CIS is one of the few who provides proper quality guides and they definitely help many companies around the world.\nBeside the benchmarks and embedded scripts, CIS has their own auditing tool. Unfortunately, this is not a free download. With the right membership, you are entitled to download their tooling. For smaller companies, this license is pretty expensive though. We clearly love open source, so let\u0026rsquo;s have a look at some alternatives!\nFocus on Automation Companies want to be more agile, using cloud technology, automation tools for configuration. Still, they often forget to implement proper system hardening. As an after-thought, hardening guides are then used to \u0026ldquo;fix\u0026rdquo; the security gaps on the system.\nOpenSCAP Another great way opposed to manuals and guides is the usage of SCAP (Security Content Automation Protocol) or more specifically OpenSCAP. This open source software helps with automated testing of security controls. While this is a great step in the right direction, there are still some flaws though.\nThe biggest issues with SCAP are, portability, easy of use and supported platforms. Each \u0026ldquo;checklist\u0026rdquo; has to be defined in a policy document. While everything is available as XML based documents, the format and structure is not really friendly for the average user. Easy of use is definitely a characteristic we value high in software solutions.\nAuditing Another alternative to SCAP is the usage of specialized auditing tools, like our own open source tool Lynis . It is available for free and no installation is needed. Lynis has been extensively tested on Linux, BSD, macOS, and other Unix-based platforms.\nBenefits The big benefit of using an auditing solution is the focus on continuous auditing. This results in improving your environment step by step. Security is not a product, but a delicate process. Instead of doing a one-time hardening exercise, it\u0026rsquo;s better to look for improvement all year round. This way of working clearly will result in better security defenses in the long run.\nConclusion We love CIS benchmarks, hardening guides and security tips. However, they are time-consuming and we love to save time where we can. OpenSCAP is a great alternative, however only works when all Linux distributions would properly embed it by default. If you have the right platform, it might be good fit for your environment.\nWe believe that for most companies specialized auditing tools are the best option available. That is, when your goal is to secure your IT environment. It is the most extensive and quickest method to perform a security audit. Continuously monitoring your environment is better than \u0026ldquo;on and off\u0026rdquo; projects trying to improve your security defenses.\n","permalink":"https://linux-audit.com/using-open-source-auditing-tools-as-alternative-for-cis-benchmarks/","tags":["auditing","guide","hardening","openscap","scap"],"title":"Using Open Source Auditing Tools as alternative to CIS Benchmarks"},{"categories":["Hardening","Web"],"contents":"Protecting against the POODLE attack\nThe POODLE attack has entered the news a few times now. The issue behind the POODLE attack is serious, as it abuses a weakness in the protocol, not the implementation. This means the only proper fix is abandoning the SSLv3 protocol and use the newer TLS protocols.\nDisable SSLv2 and SSLv3 Lighttpd commonly has its configuration settings stored in /etc/lighttpd/lighttpd.conf . Open this file and add the following two statements, to disable both protocols:\nssl.use-sslv2 = \u0026ldquo;disable\u0026rdquo;\nssl.use-sslv3 = \u0026ldquo;disable\u0026rdquo;\nNow restart the http daemon, so the new settings are applied.\nTesting SSLv3 support If your system is available via the internet, you can use the great service of SSL labs . If not, you can test your local webserver if it support SSL version 3 with OpenSSL:\necho \u0026quot;GET /\u0026quot; | openssl s_client -connect localhost:443 -ssl3 2\u0026gt; /dev/null | grep Protocol\nThis command will send a HTTP command to retrieve the home page. It uses OpenSSL to create the HTTPS connection (port 443) and forces it to use SSLv3. If it succeeds, it will show the line \u0026ldquo;Protocol : SSLv3\u0026rdquo;. In that case your server still support version 3 of the weak SSL protocol.\n","permalink":"https://linux-audit.com/disable-sslv3-lighttpd-to-protect-against-poodle-attack/","tags":["ssl","web"],"title":"Disable SSLv3 in Lighttpd to protect against POODLE attack"},{"categories":["Auditing","Hardening"],"contents":"Why setuid? Binaries with the setuid bit enabled, are being executed as if they were running under the context of the root user. This enables normal (non-privileged) users to use special privileges, like opening sockets. While this seems unnecessary for a normal user, it is actually needed for simple commands like ping.\nFinding files with setuid bit To discover all files with the setuid bit, we can use the find command. Depending on the distribution, you can use some specific parameters and special options. For example on Linux you can use -perm with slash notation (e.g. /4000). This means that if any of the file permission bits match, the result will be displayed. However, this option does not work for BSD systems, like NetBSD.\nExact match One of the best alternatives we discovered is using the -perm parameter with the octal value. However, just providing the value, would mean we have to search for the specific mode (like 4555 in the example below).\nThis exact match can be useful to fix files which got incorrect permissions and are very specific. In our case this is not the case. We want all files with the setuid bit set, which means effectively \u0026ldquo;4***\u0026rdquo;. To get this type of search, we can add a dash before the octal mode. This will also match the file if the first bit is found.\nAs can be seen in the example, the file rcmd will match. However, instead of using -4555, we can simplify the search to -4000. The zeros tell the find command that any of the values are fine for the other permission bits. So it will also include files which have normally 755 (or 4755).\n# find /bin -perm -4000 /bin/rcmd Exclude other devices/mounts Another useful addition to discovering the right binaries, is searching from the root. We are interested in directories like /bin, /sbin and /usr/(s)bin. Since we are not interested in files from other file systems mounted below /, we can exclude those first. This is done with the -xdev parameter.\nSpecific user Now we want just the files owned by the root user. Files with root as owner in combination with setuid, are executed with root privileges. All other files are not interesting. So for to be true, we add the -user root parameter.\nSetgid bit as well To complete our search, we also want to discover files which have the similar setgid bit set. This would execute files with the permission of the group. We can do this with a logical \u0026ldquo;or\u0026rdquo; statement. So we want files with the first bit to be 4 or 2.\nfind / -xdev -user root \\( -perm -4000 -o -perm -2000 \\)\nThis is one of the quickest ways to search through the file system, skipping any files which are not owned by the root user and skipping device files.\nWhat to do with the results? Most systems will reveal a few files with the setuid or setgid bit set. So having a few on your system is not an issue, but still room for improvement. Let\u0026rsquo;s have a look at the options:\nRemove the package Sometimes we come across files which we simply don\u0026rsquo;t need on our system.\nOperating Systems Command Debian dpkg -s or pkg-query -S /path/to/binary Ubuntu dpkg -s or pkg-query -S /path/to/binary Red Hat rpm -qf /path/to/binary For Debian based systems with the dpkg utility, the output looks like this:\n# dpkg -S /bin/mount mount: /bin/mount Remove the setuid bit Another logical option is removing the bit from the system. For example when the system has no normal users, why allow any software to use special rights? With chmod we can strip this bit off.\nchmod u-s /path/to/binary\nFor files that also have the setgid bit set, clear it using chmod as well.\nchmod g-s /path/to/binary\nLinux: Monitor usage with auditd If you don\u0026rsquo;t want to alter your system yet, another option would be to add the system to a Linux audit rule. This way we can track the the usage of the file.\nAn example to monitor the execution of binaries is with the follow Linux audit rule:\n-a always,exit -F path=/bin/ps -F path=/bin/ls -F perm=x -k binaries\nThis rule will monitor files /bin/ps and /bin/ls and trigger an event when being executed (perm=x), with the tag binaries (k=binaries). For more information about auditing with the Linux Audit Framework, have a look at our previous post: Configuring and auditing Linux systems with Audit daemon\n","permalink":"https://linux-audit.com/finding-setuid-binaries-on-linux-and-bsd/","tags":["audit","binaries","setuid"],"title":"Finding setuid binaries on Linux and BSD"},{"categories":null,"contents":"Want to share feedback or just want to know more about a particular article or topic? There are a few ways to get in contact.\nMastodon: mboelen E-mail: blog@cisofy.com Guest bloggers In all the years, no single proposal was received with the intention to create a dedicated article without any commercial incentive. For that reason, guest blog posts are closed till further notice.\n","permalink":"https://linux-audit.com/contact/","tags":["website"],"title":"Contact details"},{"categories":["Linux","Monitoring","System Administration"],"contents":"By default Arch will install the kernel in /boot with the name vmlinuz-linux. To determine if the system is running the latest kernel, we can compare the running kernel and the one on disk.\nRunning kernel One way to determine the running kernel is with the uname command. By default installed and with the -r parameter it will provide the kernel release version.\n# uname -r 3.17.4-1-ARCH Kernel on disk Checking the latest kernel on disk is almost as easy. In this case we have to analyze the /boot/vmlinuz-linux file, which is the default file name for the Linux kernel on Arch Linux.\nThe file utility can discover the contents of the file and determine that is indeed the kernel.\n# file /boot/vmlinuz-linux /boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version **3.17.4-1-ARCH** (builduser@tobias) #1 SMP PREEMPT Fri Nov 21 21:1, RO-rootFS, swap_dev 0x3, Normal VGA The interesting part in this case is the kernel version, which is behind the \u0026ldquo;version\u0026rdquo; keyword. In this case both the running kernel and kernel on disk are the same, so no reboot is needed.\nMonitoring If you want to automate the reboot check, we can parse the output of the uname and file commands. The small snippet below will help in performing the related check.\n#!/bin/sh set -o nounset NEXTLINE=0 FIND=\u0026#34;\u0026#34; for I in `file /boot/vmlinuz*`; do if [ ${NEXTLINE} -eq 1 ]; then FIND=\u0026#34;${I}\u0026#34; NEXTLINE=0 else if [ \u0026#34;${I}\u0026#34; = \u0026#34;version\u0026#34; ]; then NEXTLINE=1; fi fi done if [ ! \u0026#34;${FIND}\u0026#34; = \u0026#34;\u0026#34; ]; then CURRENT_KERNEL=`uname -r` if [ ! \u0026#34;${CURRENT_KERNEL}\u0026#34; = \u0026#34;${FIND}\u0026#34; ]; then echo \u0026#34;Boot required\u0026#34; fi fi Use case Testing if systems need a reboot is especially useful as part of your software patch management strategy. In our case we have embedded this test in our auditing tool Lynis, which determines for several Linux distributions if the system needs a reboot.\nUpdated (January 2015): changed script\n","permalink":"https://linux-audit.com/how-to-check-arch-linux-system-needs-reboot/","tags":["arch linux","kernel","linux","reboot"],"title":"How to check if your Arch Linux system needs a reboot"},{"categories":["Auditing","System Administration","Vulnerabilities"],"contents":"Security audit of NetBSD software packages with pkg_admin\nNetBSD is especially known for it\u0026rsquo;s diverse platforms it can run on. What is less known is the ability to audit the installed packages. In this article we have a look on how to audit NetBSD and ensure the file integrity of your packages. Performing a security audit is easy, as long as you use the right tool!\nPackages When using packages, their metadata will be installed in directory within /var/db/pkg. This tree contains information about the packages.\nnetbsd# cd /var/db/pkg netbsd# ls -l total 146 drwxr-xr-x 2 root wheel 512 Dec 3 17:23 atf-0.20 drwxr-xr-x 2 root wheel 512 Nov 24 2013 libidn-1.28 -rw-r--r-- 1 root wheel 106391 Dec 3 17:07 pkg-vulnerabilities drwxr-xr-x 2 root wheel 512 Nov 24 2013 pkg_install-20130902 -rw-r--r-- 1 root wheel 28672 Dec 3 17:23 pkgdb.byfile.db drwxr-xr-x 2 root wheel 512 Nov 24 2013 pkgin-0.6.4nb1 drwxr-xr-x 2 root wheel 512 Dec 3 17:23 shtk-1.4 drwxr-xr-x 2 root wheel 512 Dec 3 17:23 sysupgrade-1.5nb1 drwxr-xr-x 2 root wheel 512 Dec 3 17:13 wget-1.14nb3 This directory can also contain a file named pkg-vulnerabilities. This file contains information about software vulnerabilities and can be used to check what installed software packages are vulnerable.\nMoving deeper When we look into the subdirectories within /var/db/pkgs, we see a structured format of files, which include the actual metadata about the package.\n# ls -l total 78 -r--r--r-- 1 root wheel 3455 Nov 24 2013 +BUILD_INFO -r--r--r-- 1 root wheel 398 Nov 24 2013 +BUILD_VERSION -r--r--r-- 1 root wheel 46 Nov 24 2013 +COMMENT -rw-r--r-- 1 root wheel 3784 Nov 24 2013 +CONTENTS -r-xr-xr-x 1 root wheel 4075 Nov 24 2013 +DEINSTALL -r--r--r-- 1 root wheel 530 Nov 24 2013 +DESC -rwxr-xr-x 1 root wheel 9090 Nov 24 2013 +DIRS -rwxr-xr-x 1 root wheel 11075 Nov 24 2013 +FILES -rwxr-xr-x 1 root wheel 2838 Nov 24 2013 +INFO_FILES -r-xr-xr-x 1 root wheel 28793 Nov 24 2013 +INSTALL -r--r--r-- 1 root wheel 8 Nov 24 2013 +SIZE_ALL -r--r--r-- 1 root wheel 8 Nov 24 2013 +SIZE_PKG Besides normal information (like a version number), there are actually some shell scripts. Mostly they deal with the directories, files and permissions.\nInstall pkg-vulnerabilities file Before checking the system, it will need the pkg-vulnerabilities file. Installing is as easy as running the pkg_admin tool with the fetch-pkg-vulnerabilities parameter.\npkg_admin fetch-pkg-vulnerabilities\nChecking the integrity of the vulnerabilities file The pkg_admin tool is also able to check the integrity of the fetched file. Normally it should show no output, meaning everything is fine. If not, something like this shows up:\n# pkg_admin check-pkg-vulnerabilities /var/db/pkg/pkg-vulnerabilities pkg_admin: SHA1 hash doesn\u0026#39;t match Running vulnerability scan With the audit parameter we can start a vulnerability scan. It perform a security audit on the installed packages. Every package which matches a specific version, will be flagged.\nDiscovered vulnerability in wget after running audit\nIntegrity check Another thing the pkg_admin tool can perform, is an integrity check of the installed files. It uses the metadata from the packages directory and compares them with the actual files on disk.\npkg_admin discovered mismatches during file integrity check\nThis small NetBSD utility is very nifty tool and a sign that NetBSD is taking security serious as well. Happy auditing!\n","permalink":"https://linux-audit.com/perform-netbsd-security-audit-with-pkg_admin/","tags":["package manager","software","vulnerabilities"],"title":"Perform NetBSD security audit with pkg_admin"},{"categories":["Firewall"],"contents":"The usage of nftables will slowly grow in the upcoming years, with the goal to become the successor of iptables. Where iptables rules are harder to parse, nftables comes by default with an exporting facility. Exports formats include JSON and XML.\nCommand syntax When using the command line utility nft for the first time, it looks a little bit unfriendly to the user. No suggestions on what to do, nor clear help on often used commands. To save you some time, we will look into nftables and document them for easy access later on. We are sure the utilities of nftables, with nft in particular, will get some work in the upcoming releases.\nExporting rules The tool nft has an export option, followed by the format to export. Right now it support both JSON or XML. These formats are common and very easy to parse.\nExport to XML:\nnft export xml\nnft export json\nImporting nftables rules At this moment there is no import function yet. According to the documentation, this will be implemented in the upcoming releases. Clearly a useful option for sharing rules over many systems. One great example is the proposed nf**-sync** utility, which replicates nftables rules.\n","permalink":"https://linux-audit.com/networking/nftables/exporting-rules-nftables/","tags":["firewall","linux","nft","nftables"],"title":"Exporting nftables rules and configuration"},{"categories":["System Administration"],"contents":"Systemd used a binary log to store information about specific events. These events include the boot sequence and the related output. In this article we have a look at finding our boot logs in systemd journals.\nBinary logging When using systemd, boot data is stored in journals, a binary format. There is big benefit of saving boot data in a binary format: log information of each boot can be stored separately, linked to other pieces of information, and queried easier and quicker. For example, different boots can be compared, as they are individually available.\nThe journal can be queried with the journalctl command. When using the --list-boots parameter, we get a list of entries which each represent a different system boot.\nOutput of journalctl --list-boots\nIn this output the first column is the reference ID. The last entry on screen (ID equals zero) is the active boot. The utility will give each boot a (negative) number to show how many boots is was ago.\nThe second field is the boot ID. These first two IDs can be used when referring to a specific boot. Next there is the day, date, time and timezone, when the first entry entered the journal. These are then followed by the same fields, representing the last entry of the journal.\nWith these references, we can now see the details of a specific boot by using the reference ID or boot ID itself.\njournalctl --boot=fa733c82bc8c479bb1d92a964e4680e6\n","permalink":"https://linux-audit.com/systemd/finding-boot-logs-in-systemd-journals/","tags":["journalctl","systemd"],"title":"Finding boot logs in systemd journals"},{"categories":["Network","System Administration"],"contents":"System administrators and security professionals searching for listening ports on a server, are definitely familiar with the netstat command. However, newer distributions do not have the tool default installed anymore. Time to start using ss besides our beloved netstat command.\nss Socket statistics, or ss for short, is an easy replacement command for netstat. One way to use it, is with the option -a, short for all information.\nss -a This reveals a lot of information, so it might be better to tune it to something like ss -aut.\n-a: show listening and non-listening sockets -u: show UDP -t: show TCP # ss -aut Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port udp UNCONN 0 0 *:bootpc *:* tcp LISTEN 0 128 *:ssh *:* tcp ESTAB 0 0 192.168.1.251:ssh 192.168.1.220:hnmp tcp LISTEN 0 128 :::19531 :::* tcp LISTEN 0 128 :::ssh :::* This way it will show similar information to what netstat shows. When using it for very specific requests, you should refer to the man page, as it has some nice options. One of them is showing specific TCP connection state information\nDetailed TCP state information included with ss -i\nOverview of common ss options Full option Short option Usage \u0026ndash;all -a Show listening and non-listening sockets (e.g. active connections) \u0026ndash;listening -l Display only listening sockets \u0026ndash;numeric -n Do not resolve names, such as hostnames, or services \u0026ndash;processes -p Show process name \u0026ndash;tcp -t TCP sockets \u0026ndash;udp -u UDP sockets People who like to audit their system and investigate what ports are opened, can use this command as an alternative to systems without netstat. Right now most systems will have one of the tools available.\nAs these flags aren\u0026rsquo;t always easy to remember, a good tip might be to think of plants, as it reveals a good amount of information.\nss -plants\nConclusion Not many people like change. But if you like it or not, ss will be there when netstat is not. Besides that, ss has a few benefits like showing interesting new information.\n","permalink":"https://linux-audit.com/alternative-netstat-ss-tool/","tags":["netstat","ss"],"title":"Alternative for netstat: ss tool"},{"categories":["Hardening","Linux","System Administration"],"contents":"Security of Linux systems and applications can be greatly improved by using hardening measures. One of these measures is called Linux capabilities. Capabilities are supported by the kernel for some while now. Using capabilities we can strengthen applications and containers. Unfortunately, this powerful tool is still underutilized. Time to change that! This article helps to understand and apply them.\nWhat are Linux capabilities? Normally the root user (or any ID with UID of 0) gets a special treatment when running processes. The kernel and applications are usually programmed to skip the restriction of some activities when seeing this user ID. In other words, this user is allowed to do (almost) anything.\nLinux capabilities provide a subset of the available root privileges to a process. This effectively breaks up root privileges into smaller and distinctive units. Each of these units can then be independently be granted to processes. This way the full set of privileges is reduced and decreasing the risks of exploitation.\nWhy capabilities? To better understand how Linux capabilities work, let\u0026rsquo;s have a look first at the problem it tries to solve.\nLet\u0026rsquo;s assume we are running a process as a normal user. This means we are non-privileged. We can only access data that owned by us, our group, or which is marked for access by all users. At some point in time, our process needs a little bit more permissions to fulfill its duties, like opening a network socket. The problem is that normal users can not open a socket, as this requires root permissions.\nOption 1: Giving everyone root permissions One of the solutions is to allow some permissions (by default) to all users. There is a serious flaw in this approach. Allowing this kind of permissions, for all users, would open up the system for a flood of system abuse. The reason is that every small opportunity is being used for good, but also for bad. Giving away too many privileges by default will result in unauthorized changes of data, backdoors and circumventing access controls, just to name a few.\nOption 2: Using a fine-grained set of privileges For example, a web server normally runs at port 80. To start listening on one of the lower ports (\u0026lt;1024), you need root permissions. This web server daemon needs to be able to listen to port 80. However, it does not need access to kernel modules as that would be a serious threat to the integrity of the system!. Instead of giving this daemon all root permissions, we can set a capability on the related binary, like CAP_NET_BIND_SERVICE. With this specific capability, it can open up port 80. Much better!\nReplacing setuid with capabilities Assigning the setuid bit to binaries is a common way to give programs root permissions. Linux capabilities is a great alternative to reduce the usage of setuid.\nInsight: Capabilities break up root privileges in smaller units, so root access is no longer needed. Most of the binaries that have a setuid flag, can be changed to use capabilities instead.\nSee the related article [Hardening Linux binaries by removing the setuid bit][1] and apply this to your system.\nAvailable capabilities Support by the Linux kernel Linux capabilities are defined in a header file with the non-surprising name capability.h. The number of capabilities supported by recent Linux versions is close to 40. To see the highest capability number for your kernel, use the data from the /proc file system.\ncat /proc/sys/kernel/cap_last_cap\nThe full list of available Linux capabilities for the active kernel can be displayed using the capsh command.\ncapsh --print\nThe same number from the cap_last_cap file might be also displayed at the end of a capability set.\nThe capsh command shows the available Linux capabilities\nNormal users typically don\u0026rsquo;t have any capabilities assigned. This is also what we have seen in the screenshot. That is why the current list is empty. Now that changes if you switch to your root user.\nCurrent capabilities To see the capabilities for a particular process, use the status file in the /proc directory. As it provides more details, let\u0026rsquo;s limit it only to the information related to Linux capabilities.\ncat /proc/1234/status | grep Cap\nThis command should return 5 lines on most systems.\nCapInh = Inherited capabilities CapPrm = Permitted capabilities CapEff = Effective capabilities CapBnd = Bounding set CapAmb = Ambient capabilities set An explanation about these specific types will follow. Let\u0026rsquo;s start with some example output that you may get on your system.\nCapInh: 0000000000000000 CapPrm: 0000003fffffffff CapEff: 0000003fffffffff CapBnd: 0000003fffffffff CapAmb: 0000000000000000 These hexadecimal numbers don\u0026rsquo;t make sense. Using the capsh utility we can decode them into the capabilities name.\ncapsh --decode=0000003fffffffff\nAlthough that works, there is another and easier way. To see the capabilities of a running process, simply use the getpcaps tool followed by its process ID (PID). You can also provide a list of process IDs.\ngetpcaps 1234\nThe getpcaps tool uses the capget() system call to query the available capabilities for a particular thread. This system call only needs to provide the PID to obtain more information.\ncapget({_LINUX_CAPABILITY_VERSION_3, 1234}, {CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_FSETID|CAP_KILL|CAP_SETGID|CAP_SETUID|CAP_SETPCAP|CAP_LINUX_IMMUTABLE|CAP_NET_BIND_SERVICE|CAP_NET_BROADCAST|CAP_NET_ADMIN|CAP_NET_RAW|CAP_IPC_LOCK|CAP_IPC_OWNER|CAP_SYS_MODULE|CAP_SYS_RAWIO|CAP_SYS_CHROOT|CAP_SYS_PTRACE|CAP_SYS_PACCT|CAP_SYS_ADMIN|CAP_SYS_BOOT|CAP_SYS_NICE|CAP_SYS_RESOURCE|CAP_SYS_TIME|CAP_SYS_TTY_CONFIG|CAP_MKNOD|CAP_LEASE|CAP_AUDIT_WRITE|CAP_AUDIT_CONTROL|CAP_SETFCAP|CAP_MAC_OVERRIDE|CAP_MAC_ADMIN|CAP_SYSLOG|CAP_WAKE_ALARM|CAP_BLOCK_SUSPEND|CAP_AUDIT_READ, CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_FSETID|CAP_KILL|CAP_SETGID|CAP_SETUID|CAP_SETPCAP|CAP_LINUX_IMMUTABLE|CAP_NET_BIND_SERVICE|CAP_NET_BROADCAST|CAP_NET_ADMIN|CAP_NET_RAW|CAP_IPC_LOCK|CAP_IPC_OWNER|CAP_SYS_MODULE|CAP_SYS_RAWIO|CAP_SYS_CHROOT|CAP_SYS_PTRACE|CAP_SYS_PACCT|CAP_SYS_ADMIN|CAP_SYS_BOOT|CAP_SYS_NICE|CAP_SYS_RESOURCE|CAP_SYS_TIME|CAP_SYS_TTY_CONFIG|CAP_MKNOD|CAP_LEASE|CAP_AUDIT_WRITE|CAP_AUDIT_CONTROL|CAP_SETFCAP|CAP_MAC_OVERRIDE|CAP_MAC_ADMIN|CAP_SYSLOG|CAP_WAKE_ALARM|CAP_BLOCK_SUSPEND|CAP_AUDIT_READ, 0}) = 0\nIn this output, we see also version 3 of the capabilities. This was added since Linux version 2.6.26.\nIt is also interesting to see the capabilities of a set of processes that have a relationship.\ngetpcaps $(pgrep nginx)\nIf you run this on a system with nginx, you will see something special. The PID of the master process has capabilities, while the child processes or workers have none. This is because only the master requires the special permissions, like listening to a network port. The child processes then can do the work, like answering HTTP requests.\nCapability bounding set The capability bounding set defines the upper level of available capabilities. During the time a process runs, no capabilities can be added to this list. Only the capabilities in the bounding set can be added to the inheritable set, which uses the capset() system call. If a capability is dropped from the boundary set, that process or its children can no longer have access to it.\nCapabilities overview Let\u0026rsquo;s have a look at some of the available capabilities and what they do.\nCAP_CHOWN If you ever changed the owner of a file, you will be familiar with the chown command. This capability provides the privilege to do this. It allows changing both the owner as the group. Good to know is that this only applies when _POSIX_CHOWN_RESTRICTED is active, which is true on most Linux systems. By using the getconf command we can validate this.\ngetconf -a | grep _POSIX_CHOWN_RESTRICTED\nLimiting the capabilities for processes You can test what happens when a particular capability is dropped by using the capsh utility. This is a way to see what capabilities a particular program may need to function correctly. The capsh command can run a particular process and restrict the set of available capabilities.\nRun the same command with one single ping to our local system.\ncapsh --print -- -c \u0026quot;/bin/ping -c 1 localhost\u0026quot;\nDropping capabilities with capsh If we drop the CAP_NET_RAW capabilities for ping, then the ping utility should no longer work.\ncapsh --drop=cap_net_raw --print -- -c \u0026quot;/bin/ping -c 1 localhost\u0026quot;\nBesides the output of capsh itself, the ping command itself should also raise an error.\nping: icmp open socket: Operation not permitted\nThe error clearly shows that the ping command is not allowed to open an ICMP socket. Now we know for sure that this works as expected.\nBinaries with setuid bit Capabilities are a great way to replace binaries with the setuid bit set. This special bit gives users full root permissions under the context of that process. As you can imagine, if the program contains a flaw, the non-privileged user can \u0026ldquo;break out\u0026rdquo; and become the equivalent of the root user.\nStill many Linux distributions use the setuid on several binaries, while capabilities can replace the bit.\nTesting Linux capabilities To test and play with Linux capabilities, have a look at the captest command. This tool shows its own capabilities, but also tries to do privilege escalation. By applying limitations using the capsh command, you can test what the impact is.\nUseful tools and commands All commands related to Linux capabilities Command Description capsh capsh provides a capability shell wrapper to test Linux capabilities captest captest performs a set of tests related to capabilities and demonstrates them filecap filecap shows available capabilities set on binaries in $PATH or specified directory firejail firejail sandboxes applications getcap getcap queries the available file capabilities getpcaps getpcaps shows the available process capabilities netcap netcap shows an overview of network-related processes and their capabilities pscap pscap shows an overview of processes and their assigned capabilities setcap setcap adds or removes available file capabilities Conclusion Capabilities are a great way to split up root permissions and hand out some permissions to non-privileged users. Unfortunately, still many binaries have the setuid bit set, while they should be replaced with capabilities instead.\nHave a look at the overview of Linux capabilities to learn more.\n","permalink":"https://linux-audit.com/kernel/capabilities/linux-capabilities-101/","tags":["capabilities","capsh","linux","pgrep","setuid","tutorial"],"title":"Linux capabilities 101"},{"categories":["Development","Hardening","Software"],"contents":"Hardening the kernel with kernel.yama.ptrace_scope\nPtrace is a great troubleshooting tool for developers to determine how a process functions. It can be used to find programming flaws, like memory leakage. On the other hand, the tool also be used by people with malicious intent. For example to debug a process as a non-privileged user and find the contents of application memory.\nYama Linux has the ability to include Linux Security Modules, to provide additional features with the means of a module. Yama does Discretionary Access Control of some kernel related functions, like defining if process tracing (ptrace) is allowed.\nkernel.yama.ptrace_scope The parameter kernel.yama.ptrace_scope helps system administrators to select what processes can be debugged with ptrace.\nWe can determine the active value with sysctl or using the pseudo file system /proc and find the related key.\n# sysctl kernel.yama.ptrace_scope kernel.yama.ptrace_scope = 1 Or query the /proc file system.\n# cat /proc/sys/kernel/yama/ptrace_scope 1 For this particular key there are four valid options: 0-3\nkernel.yama.ptrace_scope value Meaning 0 All processes can be debugged, as long as they have same uid. This is the classical way of how ptracing worked. 1 Only a parent process can be debugged. 2 Only admin can use ptrace, as it required CAP_SYS_PTRACE capability. 3 No processes may be traced with ptrace. Once set, a reboot is needed to enable ptracing again. Using Ptrace - Advice Yama LSM enforcing enabled\ngoogle-sandbox-with-yama-lsm-enforcing.png\nServers If your system is running in the DMZ and processes high sensitive data, there is usually no reason to allow ptrace at all. Best is to disable it completely (kernel.yama.ptrace_scope = 3).\nFor servers in general you might want to apply rule, or choose a slightly less restrictive value (2 or 1).\nDesktops On desktop systems where you are the only user, can have a less restricted option (2, 1 or even disabled).\nThe Yama LSM is also used in Google Chrome, as can been seen in the related screenshot.\nIf you want to dive deeper into the details of this LSM, have a look at Yama Linux Security Module .\n","permalink":"https://linux-audit.com/protect-ptrace-processes-kernel-yama-ptrace_scope/","tags":["kernel","linux","proc","procfs","sysctl"],"title":"Protect against ptrace of processes: kernel.yama.ptrace_scope"},{"categories":["System Administration"],"contents":"Solving failed units with systemctl Systemd is an alternative service manager to the more traditional init system. To ensure the system is healthy, failed units should be investigated on a regular basis. Sooner or later a unit might fail and showing up the systemctl listing. In this article we have a look at how to solve it.\nWhy do services fail? During the start of the system, enabled services are started and queued to be executed. Most processes will start correctly and systemd logs the related status in the journal. However, in some cases a service might enter a \u0026ldquo;failed state\u0026rdquo;, as a result of another command not finishing properly.\n# systemctl UNIT LOAD ACTIVE SUB DESCRIPTION -.mount loaded active mounted / boot.mount loaded active mounted /boot dev-hugepages.mount loaded active mounted Huge Pages File System ● dev-mqueue.mount loaded failed failed POSIX Message Queue File System run-user-0.mount loaded active mounted /run/user/0 sys-kernel-config.mount loaded active mounted Configuration File System sys-kernel-debug.mount loaded active mounted Debug File System tmp.mount loaded active mounted Temporary Directory Services usually fail because of a missing dependency (e.g. a file or mount point), missing configuration, or incorrect permissions. In this example we see that the dev-mqueue unit with type mount fails. As the type is a mount, the reason is most likely because mounting a particular partition failed.\nBy using the systemctl status command we can see the details of the dev-mqueue.mount unit:\n# systemctl status dev-mqueue.mount ● dev-mqueue.mount - POSIX Message Queue File System Loaded: loaded (/usr/lib/systemd/system/dev-mqueue.mount; static) Active: failed (Result: exit-code) since Sun 2014-11-23 17:53:10 CET; 4min 12s ago Where: /dev/mqueue What: mqueue Docs: man:mq_overview(7) http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Process: 446 ExecMount=/bin/mount -n mqueue /dev/mqueue -t mqueue (code=exited, status=32) Nov 23 17:53:10 localhost.localdomain systemd[1]: dev-mqueue.mount mount process exited, code=exited status=32 Nov 23 17:53:10 localhost.localdomain systemd[1]: Failed to mount POSIX Message Queue File System. Nov 23 17:53:10 localhost.localdomain systemd[1]: Unit dev-mqueue.mount entered failed state. This information shows the related command which was executed. We see the unit failed on exit-code as it was not the expected value of 0 (actually it is 32). Manually running the command shows the device /dev/mqueue is missing.\nSimilar to this service, IPMI fails on our virtual machine. As there is no /dev/ipmi* device, the service can\u0026rsquo;t start and fails:\n# systemctl status ipmievd.service ? ipmievd.service - Ipmievd Daemon Loaded: loaded (/usr/lib/systemd/system/ipmievd.service; enabled) Active: failed (Result: exit-code) since Sun 2014-11-23 16:08:48 CET; 1h 36min ago Process: 550 ExecStart=/usr/sbin/ipmievd $IPMIEVD_OPTIONS (code=exited, status=1/FAILURE) Nov 23 16:08:47 localhost.localdomain ipmievd[550]: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Nov 23 16:08:47 localhost.localdomain ipmievd[550]: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Nov 23 16:08:47 localhost.localdomain ipmievd[550]: ipmievd: using pidfile /var/run/ipmievd.pid0 Nov 23 16:08:47 localhost.localdomain ipmievd[550]: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Nov 23 16:08:47 localhost.localdomain ipmievd[550]: Unable to open interface Nov 23 16:08:48 localhost.localdomain systemd[1]: ipmievd.service: control process exited, code=exited status=1 Nov 23 16:08:48 localhost.localdomain systemd[1]: Failed to start Ipmievd Daemon. Nov 23 16:08:48 localhost.localdomain systemd[1]: Unit ipmievd.service entered failed state. Nov 23 16:08:48 localhost.localdomain systemd[1]: ipmievd.service failed. Clearing failed units You can manually clear out failed units with the systemctl reset-failed command. This can be done for all units, or a single one.\nServices which are no longer needed, are better to be stopped and disabled.\nsystemctl stop rngd.service\nsystemctl disable rngd.service\nThat\u0026rsquo;s all!\n","permalink":"https://linux-audit.com/systemd/auditing-systemd-solving-failed-units-with-systemctl/","tags":["systemd","troubleshooting"],"title":"Auditing systemd: solving failed units with systemctl"},{"categories":["Hardening","Network"],"contents":"Most Linux system administrators will be familiar with iptables on Linux. Less known is the arptables utility, which controls filtering ARP packets.\nInstallation of arptables The arptables utility is easy to set-up, as the main functionality is already implemented in the Linux kernel. Just install the arptables package on your favorite Linux distribution.\nDebian / Ubuntu: apt install arptables Red Hat: yum install arptables Configuration example To show the effect of filtering traffic, we will show an example by filtering router traffic and blocking it. This way we won\u0026rsquo;t be able to connect to the internet.\nWith the arp command we can query the current list of known ARP addresses.\n# arp Address HWtype HWaddress Flags Mask Iface System.cisofy.com ether 00:a7:22:23:d1:f3 C eth0 Router.cisofy.com ether d8:d7:21:22:5a:8d C eth0 Arptables can block traffic by filtering out the IP. So let\u0026rsquo;s query the arp list again, now in numeric format.\n# arp -n Address HWtype HWaddress Flags Mask Iface 192.168.1.20 ether 00:a7:22:23:d1:f3 C eth0 192.168.1.1 ether d8:d7:21:22:5a:f4 C eth0 Time to block the router (192.168.1.1):\narptables -A INPUT -s 192.168.1.1 -j DROP\nSo we dropped traffic to this IP adress, right? Let\u0026rsquo;s try!\n# ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.645 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.370 ms ^C --- 192.168.1.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.370/0.507/0.645/0.139 ms Well, that didn\u0026rsquo;t work like intended. We dropped ARP related traffic to the IP address, but not on IP level. This is also visible in the arp -n list:\n# arp -n Address HWtype HWaddress Flags Mask Iface 192.168.1.20 ether 00:a7:22:23:d1:f3 C eth0 192.168.1.1 ether d8:d7:21:22:5a:f4 C eth0 So to make this work, we simply have to flush the ARP cache. We delete the related ARP entry:\n# arp -d 192.168.1.1 # arp -n Address HWtype HWaddress Flags Mask Iface 192.168.1.20 ether 00:a7:22:23:d1:f3 C eth0 192.168.1.1 (incomplete) eth0 The arp utility will show an incomplete entry. It knows that recently some traffic passed by, but the MAC address is unknown.\nLet\u0026rsquo;s ping again:\n# ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. From 192.168.1.21 icmp_seq=1 Destination Host Unreachable From 192.168.1.21 icmp_seq=2 Destination Host Unreachable That looks better!\nSpecific traffic filtering Back to our original mission: only allow our router to exchange ARP packets. Block ARP traffic from all machines (default: DENY)\narptables -P INPUT DROP\nAllow the router by defining an ACCEPT statement with a fixed ARP address.\narptables -A INPUT --source-mac d8:d7:21:22:5a:f4 -j ACCEPT\nAll ARP packets are blocked now. Each system which will transmitting traffic will end up as an (incomplete) entry.\nEnable all ARP traffic If we want to allow traffic again:\narptables -P INPUT ACCEPT\nThen flush.\narptables --flush\nFlushing the full ARP cache can also be done with ip utility:\n`ip -s neighbour flush all```\nConclusion Arptables is a very powerful utility to filter traffic and avoid an unexpected router taking over our connectivity. However, keep in mind that connectivity is not fully blocked. Only ARP traffic is blocked (layer 2/3 on the OSI model). If someone is able to manually add an entry to the ARP table, traffic is able to flow again.\n","permalink":"https://linux-audit.com/networking/filtering-arp-traffic-with-linux-arptables/","tags":["arp","arptables","ip","network"],"title":"Filtering ARP traffic with Linux arptables"},{"categories":["Network","System Administration"],"contents":"There are several reasons when you might need to clear your ARP cache. There are two common ways on Linux systems, typically using the arp or ip utility. Depending on your Linux distribution and the availability, we suggest using the ip tool.\nClearing cache with ip Newer Linux distributions have the ip utility. The ip tool has a more advanced way to clear out the full ARP cache.\nip -s -s neigh flush all\nThe first -s will provide a more verbose output. By adding one more, we can select the neighbor table. The neighbor table with the ip command equals both the ARP and NDISC cache. Note that the -s options are not available on all versions of the ip command. If it not supported for your version of ip, then simply remove them from the command.\nThe output of the flush all command will produce the following output.\nThe ARP cache is cleared, with verbose output\nClearing cache with arp command The arp utility does not accept an option to clear the full cache. Instead, it allows to flush out entries found with the -d option.\narp -d 192.168.1.1\nAfter deleting, have a look with the arp utility again to see the new list:\narp -n\nThe output of this command will typically show the active ARP entries.\nAddress HWtype HWaddress Flags Mask Iface 192.168.1.1 (incomplete) eth0 192.168.1.2 ether 00:02:9b:a2:d3:f3 C eth0 192.168.1.3 ether 00:02:9b:d9:d1:a2 C eth0 The 192.168.1.1 entry now shows as incomplete, which means the ARP entry will be refreshed when it is needed again.\nConclusion Depending on your distribution, the ip utility is quicker if you want to flush out the full ARP cache. For individual entries, the arp tool will do the job as quickly. Both tools are available for most distributions, including Arch Linux, CentOS, Debian, Fedora, RHEL, and Ubuntu.\nDid this article to clear the ARP cache help you as well? Wonderful! Become part of the community and share this on social media to let others know. Got questions or suggestions? Let it know!\n","permalink":"https://linux-audit.com/how-to-clear-the-arp-cache-on-linux/","tags":["arp","ip","network"],"title":"How to clear the ARP cache on Linux?"},{"categories":["Access Control List","File Systems"],"contents":"What are extended attributes? Extended attributes or xattrs, are an extensible mechanism to store metadata on a filesystem. Metadata is a collection of information or data points about a particular object. If we would compare this article, the metadata contains the title, author, description, language, Twitter image, etc.\nNormally the file system can only store a limited set of information about files. Typically this is the filename, ownership, file permissions, and dates. By using extended attributes, we can describe more properties of the file.\nSupport for extended attributes Not all file systems have support for xattrs. However, the popular ones do, like EXT4, Btrfs, ReiserFS, JFS, and ZFS. To determine if your file system has xattr support enabled, check the options file of the related device:\n# cat /proc/fs/ext4/sda1/options | grep xattr user_xattr One way to set an attribute for a file is by adding an access control list (ACL). This can be done with the setfacl command. For example, we can allow the web server daemon to read data from /data/storage.\nsetfacl -m u:www-data:r /data/storage\nRunning the command won\u0026rsquo;t give any output. So let\u0026rsquo;s check if something has changed:\n# ls -l total 4 drwxr-xr-x**+** 2 root root 4096 Nov 18 16:00 storage The plus sign in ls reveals there is something different than the other files. This is because of adding the extended attribute.\nAlthough we could use the getfacl command to determine the permissions, we can actually use the getfattr command to see what kind of attribute is added.\n# getfattr /data/storage getfattr: Removing leading \u0026#39;/\u0026#39; from absolute path names # file: data/storage system.posix_acl_access Now we know for sure it is an ACL stored in the extended attributes of this particular file (or actually directory).\nIf we want to see detailed information, we can use the xattr tool for that.\nUsing xattr to list extended attributes of a file\nOther attributes security.capability The security.capability files stores Linux capabilities for the related file. Applies to binaries which are provided one or more capabilities via this file.\nsecurity.ima For the Integrity Measurement Architecture (IMA), the file security.ima stores a hash or digital signature.\nsecurity.evm Similar to security.ima, the Extended Verification Module (EVM) stores a hash/HMAC or digital signature in this file. The different with IMA is that it protects the metadata of the file, not the contents.\nRelated tools getfacl Installation: apt install acl\ngetfattr Installation: apt install attr\nxattr Installation: apt install xattr\n","permalink":"https://linux-audit.com/using-xattrs-extended-attributes-on-linux/","tags":["ACL","file system","getfacl","setfacl"],"title":"Using xattrs or Extended Attributes on Linux"},{"categories":["Hardening","System Administration"],"contents":"Hardening Linux binaries by removing setuid\nNormally Unix based systems use two kind of processes: privileged and unprivileged. The first category is usually used for administrative purposes, like starting and stopping other processes, tuning the kernel and opening sockets.\nRoot permissions The command ping is a great example why even small programs needs root permissions. In a first glance you might consider this tool to be simple: send a package to a host and see if it responds. The truth is that a network socket needs to be opened, to send an ICMP package.\nLet\u0026rsquo;s have a look at the ping binary:\n$ ls -l /bin/ping -rwsr-xr-x 1 root root 44168 May 7 23:51 /bin/ping So the binary itself is small in size. It actually turns red in our console. This is due to the setuid bit, which can be seen due to the small s (where normally the \u0026ldquo;x\u0026rdquo; is).\nNext step is to remove the setuid bit from the binary and see the result:\n$ sudo chmod u-s /bin/ping $ ls -l /bin/ping -rwxr-xr-x 1 root root 44168 May 7 23:51 /bin/ping Removing the setuid bit has turned the binary into green, which is the common color for executable files and binaries.\nNow let\u0026rsquo;s try using the ping command:\n$ ping cisofy.com ping: icmp open socket: Operation not permitted So that clearly shows that our beloved ping command is not longer working. For a system without normal users this would be great regarding hardening. The less setuid binaries, the better.\nIn this case we assume we want normal users to still use the ping command. So let\u0026rsquo;s add a capability which allows to open up a network based socket. Look in the man page we find the cap_net_raw capability:\nCAP_NET_RAW\nuse RAW and PACKET sockets; bind to any address for transparent proxying. Great, this seems exactly what we want!\nCapability sets Now the challenge is understanding how to apply the capability the proper way. Just slapping the capability onto the binary is not enough, we will also need to define how it applies. This is done via a capability sets.\nEach process thread has three capability sets, which may contain some, all or none of the following sets. From the man pages:\nEffective - the capabilities used by the kernel to perform permission checks for the thread. Permitted - the capabilities that the thread may assume (i.e., a limiting superset for the effective and inheritable sets). If a thread drops a capability from its permitted set, it can never re-acquire that capability (unless it exec()s a set-user-ID-root program). Inheritable - the capabilities preserved across an execve(2). A child created via fork(2) inherits copies of its parent\u0026rsquo;s capability sets. See below for a discussion of the treatment of capabilities during exec(). Using capset(2), a thread may manipulate its own capability sets, or, if it has the CAP_SETPCAP capability, those of a thread in another process. The effective set is needed when performing a specific system call in which it needs to have a specific capability.\nSo this means for a normal binary, which will not create child processes, the permitted will do. This means at best it will be able to use the capability. It may be a limited superset of what inheritable and effective will provide. For processes which fork other processes, we might need to inherit the capabilities. In that case, use the inheritable set.\nSetting capabilities So let\u0026rsquo;s use this knowledge and apply it to the binary:\n$ sudo setcap cap_net_raw+p /bin/ping\nThen check if we have the new capability.\n/bin/ping = cap_net_raw+p We set the capability with setcap and tell it to set the permitted set. Now we test again and the ping command works like it did before.\n$ ping cisofy.com PING cisofy.com (149.210.134.182) 56(84) bytes of data. 64 bytes from cisofy.com (149.210.134.182): icmp_seq=1 ttl=58 time=10.5 ms 64 bytes from cisofy.com (149.210.134.182): icmp_seq=2 ttl=58 time=11.8 ms Why not use setuid? Usually it makes sense to allow a trusted binary to use root permissions to execute. The unfortunate thing with software is that it may contain bugs. So even the smallest mistake with a setuid binary may result in total compromise.\nUsing capabilities gives a binary root permissions for only a limited set of systems calls. So even if there would be a software leak, it may not even be abused. For example the ping command in this article. This command would not be able to change ownership of a file, with the chown system call, where a setuid binary would have been.\nWant to learn more about capabilities? Have a look at Linux capabilities 101.\n","permalink":"https://linux-audit.com/kernel/capabilities/linux-capabilities-hardening-linux-binaries-by-removing-setuid/","tags":["capabilities","file permissions","linux","setuid"],"title":"Linux Capabilities: Hardening Linux binaries by removing setuid"},{"categories":["Auditing","Compliance","PCI DSS compliance"],"contents":" A.1.2.c Verify that an entity’s users do not have write access to shared system binaries\nShared system binaries should be protected, as they form the basis of your system. PCI compliance (A.1.2.c) demands that users do not have write access to shared systems binaries. The only exception is of course the root user, so software upgrades are still possible.\nPaths for system binaries Depending on the distribution used there are several directories which have shared system binaries. Common paths are:\n/bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/libexec /usr/local/sbin /usr/local/libexec These paths can be scanned for any binary having incorrect permissions. In this particular case we are interested in binaries which can be overwritten by people in the \u0026ldquo;other\u0026rdquo; group.\nfind /bin -perm -o=w ! -type l\nThis will show any system binaries in /bin where the other group has the write bit set. We skip symlinks, as they are not interesting and give false positives to the test.\nDepending on the paths, this has to be repeated for all of them. Any findings from the find command means this binary (or file) can be written to by someone other than the owner. Usually this is a sign of bad system management or a possible intrusion.\nThis information is provided as an addition to the PCI DSS plugin for Lynis\n","permalink":"https://linux-audit.com/pci-dss-v3-linux-no-write-access-to-shared-system-binaries/","tags":["binaries","compliance","linux","pci dss"],"title":"PCI DSS Linux: No write access to shared system binaries"},{"categories":["Crypto","System Administration"],"contents":"Anyone who wants to create a new key set via GnuPG (GPG) may run into this error:\nWe need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. Not enough random bytes available. Please do some other work to give the OS a chance to collect more entropy! (Need 142 more bytes) The problem is caused by the lack of entropy (or random system noise). While trying to get more, you might keep running into this message. In our case running a find on the disk, while making sha1sums and putting that into files, was actually not enough.\nTo check the available entropy, check the kernel parameters:\ncat /proc/sys/kernel/random/entropy_avail 36 To solve the lack of entropy, we can use a random number generator utility like the rngd command\nInstallation Debian and Ubuntu apt install rng-tools\nRed Hat Enterprise Linux, Fedora, and CentOS dnf install -y rng-tools\nOther distributions most likely will have the same package name available (rng-tools).\nUsing rngd rngd -f -r /dev/urandom\nChecking the available entropy again revealed in our case a stunning 3100, almost 100 times more than before. GnuPG is now happy again and can finish creating the keys.\n","permalink":"https://linux-audit.com/gpg-key-generation-not-enough-random-bytes-available/","tags":["troubleshooting"],"title":"GPG key generation: Not enough random bytes available"},{"categories":["File systems","Hardening"],"contents":"Mount points are defined in /etc/fstab . They link a particular disk pointer to the related device (disk, partition or virtual device). By default the mount options are not focused on security, which gives us a room to further improve hardening of the system. This hardening is especially important considering our most precious data is stored here. Via mount options we can apply additional security controls to protect our data.\nMount point example Let\u0026rsquo;s have a look at our /etc/fstab file.\nExample output:\n# \u0026lt;file system\u0026gt; \u0026lt;mount point\u0026gt; \u0026lt;type\u0026gt; \u0026lt;options\u0026gt; \u0026lt;dump\u0026gt; \u0026lt;pass\u0026gt; proc /proc proc defaults 0 0 In the options column, the related mount options are defined. In this particular case it has \u0026ldquo;defaults\u0026rdquo; for /proc, meaning the options rw, suid, dev, exec, auto, nouser, and async are set.\nMount options When looking at the options, here are a few common values:\nMount option Meaning rw Read and write allowed auto Mount automatically nouser Do not allow a user to mount the file system async Asynchronous saving of data, to improve performance Since this is a virtual file system, which has no user data or binaries stored, we leave it with the defaults option.\nMount options for hardening Regarding the remaining options (suid, dev, exec), we will have a look at their \u0026ldquo;negative\u0026rdquo; opposites, to show how we can apply them to harden the system.\nnodev This option describes that device files are not allowed, like block or character devices. Normally these are only found under /dev and not seen on other mount points. Most mount points will work correctly when these are disabled, with the root file system as an exception.\nUseful for: /boot /dev/shm /home /tmp /var and data partitions\nNot suitable for: root (/)\nnoexec With this option set, binaries can\u0026rsquo;t be directly executed.\nUseful for: /boot /dev/shm /var and data partitions.\nNot suitable for: root (/), /home (when using steam, wine or development) and /tmp (e.g. compiling applications might break)\nnosuid Do not use set-user-identifier (SETUID) or set-group-identifier (SETGID) bits to take effect. These bits are set with chmod (u+s, g+s) or unset (u-s, g-s) to allow a binary running under a specific user, which is not the active user itself. For example, to allow a normal user to run the ping command with root privileges. This is needed to allow opening a socket.\nUseful for: /boot /dev/shm /home /tmp /var and data partitions\nNot suitable for: root (/)\nApply system hardening To harden mount points, replace the defaults entry and add the related options to the related field. When applying multiple options, separate them with a comma.\n","permalink":"https://linux-audit.com/securing-mount-points-on-linux/","tags":["binaries","file system","hardening","mount"],"title":"Securing mount points on Linux"},{"categories":["Auditing","Automation","Hardening","Lynis","Monitoring","Software","System Administration"],"contents":"Many people used Bastille Linux to harden their Linux systems. Unfortunately the website of Bastille seems very outdated, including the tool. This resulted in people searching for a great alternative to replace this tool. We found the alternative by actually combining different solutions, being more powerful. Security automation is hot, so forget Bastille and do it the right way.\nAutomatic hardening makes sense Most system administrators can\u0026rsquo;t keep up with the new technologies and security threats. It is simply to much to investigate everything and stay up-to-date with the latest software. Besides that, the existing systems often need management, even years after the initial software was released.\nAutomatic hardening, or security automation, reduces the effort on the part of the system administrator. The tooling is ready to go and can tighten up security controls on the related system. The big benefit is that running a tool is quick, does not require much knowledge and at least provides additional protection.\nWhy it does not.. Tools make people lazy, sometimes even uneducated. Without understanding why a change has to be implemented, it might give a false sense of security. Then there is the risk of crippling the system by implementing a security control without proper testing. Additionally there is the fact that most systems are not equal, so exceptions might be applicable or needed.\nThe alternative to Bastille Linux Lynis is not a hardening tool, but helps with hardening. Instead of just changing configuration files, it will perform an in-depth audit of the system and show the related findings. The administrator then can determine what controls are appropriate to be applied and create a custom automation script. This can be done via a normal shell script, or by using configuration automation tools like CFEngine or Puppet.\nThe big benefit of using an auditing tool is the flexibility and support for different operating systems. Often companies use different Linux distributions, resulting in a tool only support one or another. When it comes to hardening, each system has it\u0026rsquo;s own minor differences.\nSecurity automation By combining the auditing and configuration automation, we have security automation with continuous monitoring. Both the automation tool will check for inappropriate conditions and so will the auditing tool. While it initially will take a little bit more time, it will outperform the benefits of an automatic hardening tool. It will give security insights for the system administrator(s) and includes checking the configuration on a regular basis.\nFor people who are known to the Plan-Do-Check-Act cycle, they will recognize the steps. It starts with your goal for hardening and planning the initial audit (Plan), up to the implementation (Do), checking for effectiveness (Check) and act upon new findings (Act). This way of working is more in line with security, being a process and not a product. It enhances security awareness and let people act upon new findings, instead of the \u0026ldquo;fire and forget\u0026rdquo; of a tool.\nConclusion Bastille Linux is a great tool, or maybe we should say, was a great tool. Fortunately there are better alternatives nowadays, by combining tools and leverage the strengths of each tool. The combination of an auditing tool and a configuration automation tool, will provide more benefits. They include better educated personnel, more control over the implementation, continuous monitoring and working according to a process, enhancing security over time.\nDid you find a better alternative for Bastille Linux? Share it in the comments!\n","permalink":"https://linux-audit.com/alternatives-to-bastille-linux-system-hardening-with-lynis/","tags":["hardening","linux","lynis"],"title":"Alternatives to Bastille Linux: system hardening with Lynis"},{"categories":["Auditing","Compliance","PCI DSS compliance"],"contents":" A.1.2.a Verify the user ID of any application process is not a privileged user (root/admin).\nFor Unix and Linux based systems, processes should run as a non-privileged user where possible. However to be able to start, a process is usually started with root permissions (uid 0). This is required to open the required sockets (e.g. bind to port 80).\nAfter the initial start, the process drops its privileges by switching to another user. In some cases there will maintain one master process, which is started with uid 0 as well. This process is responsible for the creation of child processes, not for handling interactions with users or processes. You can consider this as an administrative process. The child processes do handle\nTo gather a list of application processes running under the context of root, we can query ps and list all related entries.\nps -ef | grep \u0026quot;^root\u0026quot;\nAnother way is to combine a few commands and only list the interesting processes, like this:\nps -ef | awk '{ if ($1==\u0026quot;root\u0026quot;) { print $8 }}' | grep -v \u0026quot;^\\[\u0026quot; | sort | uniq | grep -E -v \u0026quot;^(\\-su|awk|egrep|grep|ps|sort|uniq|su|sudo)\u0026quot;\nWith this command we query ps, filter out application processes running under the context of root and hide commands which are not interested.\nUsually this will still be a list of several items. Every process which has a master process, which have at least one process running under the root context. This is acceptable behavior as explained before. Other processes have to be analyzed by hand, to see if they are properly configured.\nThis information is provided as guidance to our PCI plugin for Lynis.\n","permalink":"https://linux-audit.com/pci-dss-v3-linux-auditing-application-processes-a-1-2-a/","tags":["auditing","awk","compliance","pci dss"],"title":"PCI DSS (v3) for Linux: Auditing application processes (A.1.2.a)"},{"categories":["Automation"],"contents":"The problem with humans is that they are smart yet slow at the same time. They can\u0026rsquo;t react to simultaneous events and aren\u0026rsquo;t always working. Besides that, they make mistakes, have to deal with budgets and internal company politics. Information security is impacted by these effects as well.\nAs you might have guessed the solution is in automation. SCAP (Security Content Automation Protocol) is one of the answers. Especially the automation part is interesting, as it can improve quality, decrease time efforts and remove the \u0026ldquo;boring\u0026rdquo; work.\nSCAP is using predefined templates, stating how a machine should look like. Not only can SCAP check a state of a configuration item, it can also push the preferred value. The problem of unsecured systems is over, right? Not really\u0026hellip;\nPros Automation is key, especially in this time where every minute equals a lot of money. SCAP is one option to automate as much as possible. Together with your configuration automation (e.g. Ansible, Cfengine, or Puppet), it can form a great team.\nSCAP already uses a predefined consensus of what is \u0026ldquo;secure\u0026rdquo;, reducing the amount of preparation work. System administrators now only have to activate the related template, run the SCAP toolkit and they are done.\nStandards like SCAP also provide a better security awareness for companies. After all, they are the experts who think about the subject and share it with the world. In this case the people from NIST and the contributors to the CIS Benchmarks.\nCons Unfortunately, SCAP has its challenges. The templates to check (and harden) systems are very specific and will only work for those operating systems, including the specific version. When running a different version, you will have to change things manually, or wait for an update.\nAchieving consensus So if your company is not the government, you will run most likely the newest versions of Linux. The policy writers of CIS and SCAP can\u0026rsquo;t keep up with that demand, as they have to research and discuss the advised best practices. They have to come to a consensus before they can draft a hardening proposal. With all the differences between Linux distributions, it is hard to come up with a clear template which works for all of them.\nDealing with exceptions Not all machines are the same, which usually results in exceptions. Such an exception might be needed due to the role a system has, the particular business owner or application running on the system. Full automation (including alteration) is not always preferred, as it might break business critical machines. That is unfortunate, as these systems benefit the most from hardening. And it are exactly these machines that need the most protection. A less intrusive tool like Lynis might be of great help here.\nConclusion Security automation is great, but we will always need people. One of the protocols to implement security automation, SCAP, is not matured enough. Right now, the combination of a good auditing tool, together with a configuration automation tool is much stronger. It saves you from the hassle of waiting for new templates, gives you ultimate flexibility and still uses a lot of automation. Combining the automation of these tools with the intellect of people and you have a much better solution.\n","permalink":"https://linux-audit.com/security-automation-for-linux-are-humans-still-needed/","tags":["automation","hardening","SCAP"],"title":"Security Automation for Linux: Are Humans Still Needed?"},{"categories":["Lynis"],"contents":"Although Lynis has many tests built-in, there are enough reasons to create your own custom tests. Instead of patching up existing files, there is a better way to run them and make use of existing functions.\nIn this article we will have a look on how to create your own tests and what functions can be used. With the software being open source and licensed under GPL, you have the flexibility to see existing tests and adjust them to your needs.\nCreating tests No custom tests enabled in Lynis\nCreating custom tests is easy, as the toolkit of Lynis is written in shell script. So there is a lot of examples available in the already existing tests. To make it even easier, there is a small template available, which is named tests_custom.template and located in the include directory.\nFirst step is to copy this file and name it tests_custom.\n# cd include # cp tests_custom.template tests_custom Within this template there is an example test defined.\n# Test : CUST-0010 # Description : Check for something interesting - template # This test first checks if OpenSSL binary was found if [ ! -z \u0026#34;${OPENSSLBINARY}\u0026#34; ]; then PREQS_MET=\u0026#34;YES\u0026#34;; else PREQS_MET=\u0026#34;NO\u0026#34;; fi Register --test-no CUST-0010 --preqs-met ${PREQS_MET} --weight L --network NO --description \u0026#34;My description\u0026#34; # Or you could use this one without any dependencies # Register --test-no CUST-0010 --weight L --network NO --description \u0026#34;My description\u0026#34; if [ ${SKIPTEST} -eq 0 ]; then FOUND=0 LogText \u0026#34;Test: checking something\u0026#34; ReportWarning ${TEST_NO} \u0026#34;Test warning\u0026#34; if [ ${FOUND} -eq 0 ]; then Display --indent 4 --text \u0026#34;- Performing custom test 1\u0026#34; --result OK --color GREEN LogText \u0026#34;Result: the test looks great!\u0026#34; else Display --indent 4 --text \u0026#34;- Performing custom test 1\u0026#34; --result WARNING --color RED LogText \u0026#34;Result: hmm bad result of this test :(\u0026#34; ReportSuggestion ${TEST_NO} \u0026#34;This could be better!\u0026#34; fi fi Prerequisites This test start with checking if the OPENSSLBINARY variable has been set (! -z means is it not zero or empty). If it is set, then the prerequisites of this particular test have been set and checking when using the Register function.\nWhen no prerequisites apply, remove the parameter from the Register function.\nRegister the test The Register function actually registers the related test and determines if there are any reasons to skip the test. If not, the variable SKIPTEST will be set to 0.\nThis function needs at least the test-no, weight, network and description.\nWith the -test-no we define an unique identifier for your test. Use always \u0026ldquo;CUST-\u0026rdquo; followed by a four-digit number.\nThe -weight defines how important the test is, using L (low), M (medium) or H (high). When unsure, use M. By using the -network parameter, you can define if it needs network access, or not. Most tests won\u0026rsquo;t use any network function, so it is safe to use \u0026ldquo;NO\u0026rdquo; as a default value.\nTips for tests From here on your test can do everything which is allowed in shell scripting. Run commands and test configuration files.\nWe suggest to avoid \u0026ldquo;exotic\u0026rdquo; commands and use as much bourne scripting (not bourne again, or bash). The reason is portability and it avoids that you have to add additional checks. When using commands which are not on the system, it may result in unexpected outcomes or errors on screen.\nMake sure to initialize any variables which you will reuse at a later stage. For example, we use COUNT and FOUND a lot. We initialize those values to zero at the beginning of each test where it is used. The reason is to avoid a false positive in a test running at a later stage.\nTesting your custom tests When you are done with adding your tests, you can easily test them by only running the custom category.\n./lynis audit systems --tests-category \u0026quot;custom\u0026quot;\nAfter running Lynis again, it should show your custom test(s) and the related result. For example after enabling the test in the template:\nOutput of custom test in Lynis\nCommon functions During the creation of your own tests, you can reuse existing functions. These will simplify the way data is logged to a log file, or displayed on screen.\nDisplay The Display function shows text on screen.\nIt uses at least the indent and text parameters. Optionally are the result and related color of the result.\nUsage: Display -indent \u0026lt;number of spaces\u0026gt; -text \u0026quot;Your personal text\u0026quot; -result \u0026quot;\u0026lt;status\u0026gt;\u0026quot; -color \u0026quot;\u0026lt;color code\u0026gt;\u0026quot;\nThe status usually is something like OK, Found, BAD, NONE, where color is one of the colors defined in the include/consts file.\nLogText This function will log a string to the defined log file.\nUsage: LogText \u0026quot;This is a test\u0026quot;\nReportException Use this function to trigger the exception message at the end of the scan, when an unexpected result occurred and you did not catch it with a normal \u0026ldquo;if\u0026rdquo; statement. For example: if you expect a test to exit with exit code 0 or 1 and it actually gave back \u0026ldquo;2\u0026rdquo;, then trigger the exception. This helps in debugging and monitoring for unexpected results.\nUsage: ReportException \u0026quot;${TEST_NO}:1\u0026quot; \u0026quot;Unexpected result from running command ps\u0026quot;\nReportSuggestion Show a suggestion on screen (at the end) and store it in the report. Additional details might be added (see include/functions).\nUsage: ReportSuggestion \u0026quot;${TEST_NO}\u0026quot; \u0026quot;We suggest to install package X\u0026quot;\nReportWarning Similar to ReportSuggestion, except that it shows the message in the warning box. Additional details might be added (see include/functions).\nUsage: ReportWarning \u0026quot;${TEST_NO\u0026quot; \u0026quot;We discovered a medium scored finding\u0026quot;\nOther functions There are more functions available. They are stored in include/functions, with a description. By searching for the related function in the existing tests, they provide many examples on how to use the function.\nGitHub Want to stay up-to-date, follow the project on GitHub: Lynis ","permalink":"https://linux-audit.com/lynis/how-to-create-custom-tests-in-lynis/","tags":["lynis"],"title":"How to create custom tests in Lynis"},{"categories":["Automation","System Administration"],"contents":"Installing from source\nSecurity automation is hot and we love it. One way is using the OpenSCAP toolkit. Unfortunately it is not mature enough, so you might want to build and install it from source. We share our findings while creating our test environment.\nInstall required components On our minimum installed CentOS 7 system, we need to install a few components. Most are related to compiling C++ and parsing XML files. Since we like to use Git, let\u0026rsquo;s start with that and obtain the source code of OpenSCAP:\nmkdir /root/openscap-build \u0026amp;\u0026amp; cd /root/openscap-build yum install git git clone https://github.com/OpenSCAP/openscap cd openscap/ Next is installing the related components to build the toolkit:\nyum install gcc yum install autoconf automake libtool yum install libcurl-devel libxml2-devel libxslt-devel pcre-devel swig yum install python-devel Optional components To support as much as possible, we want to install some additional components. They are not needed for everything, but depending on the system may be useful (e.g. RPM for CentOS).\nyum install rpm-devel libselinux-devel systemd-devel GConf2-devel\nWe skip isaconf, as this is related to Solaris.\nBuild OpenSCAP from source Time to build OpenSCAP from the source files:\nmake clean \u0026amp;\u0026amp; ./autogen.sh \u0026amp;\u0026amp; ./configure \u0026amp;\u0026amp; make\nIf everything went fine, it should end with leaving the directories and a successful compilation (something like this):\nMaking all in python3 make[3]: Entering directory `/root/openscap/openscap/swig/python3\u0026#39; make[3]: Nothing to be done for `all\u0026#39;. make[3]: Leaving directory `/root/openscap/openscap/swig/python3\u0026#39; make[3]: Entering directory `/root/openscap/openscap/swig\u0026#39; make[3]: Nothing to be done for `all-am\u0026#39;. make[3]: Leaving directory `/root/openscap/openscap/swig\u0026#39; make[2]: Leaving directory `/root/openscap/openscap/swig\u0026#39; make[2]: Entering directory `/root/openscap/openscap\u0026#39; make[2]: Leaving directory `/root/openscap/openscap\u0026#39; make[1]: Leaving directory `/root/openscap/openscap\u0026#39; So if the build was successful, we can optionally install the toolkit:\nmake install\nIn our case there are some builds between what the original CentOS 7 package provided and the newer compiled binary in /usr/local/bin:\n# /bin/oscap -V | grep oscap OpenSCAP command line tool (oscap) 1.0.3 # /usr/local/bin/oscap -V | grep oscap OpenSCAP command line tool (oscap) 1.2.0 Happy auditing!\n","permalink":"https://linux-audit.com/openscap-on-centos-7-installing-from-source/","tags":["automation","openscap"],"title":"OpenSCAP on CentOS 7 – Installing from source"},{"categories":["Auditing","Software","System Administration"],"contents":"Enhancing yum Determine available plugins and built-in security support\nTo enhance the support in our auditing tool Lynis, we wanted to know if yum supports security related functions by using a plugin or having it as built-in functionality.\nYum Yum, or Yellowdog Updater Modified, is a software management tool for Linux based systems. Usually it is used on systems running SuSE or Red Hat based (like RHEL, Fedora or CentOS). Plugins extend the functionality of yum, to improve its functionality.\nOne plugin may select the fastest software mirror, so you don\u0026rsquo;t have to benchmark them manually. Another great plugin helps with security and shows what security related updates are available. Nowadays, this functionality is built-in, as the demand for this functionality is huge.\nIn our case we want to audit the yum tool set and determine if we have the plugin available, or dealing with the built-in functions. Let\u0026rsquo;s start with the plugins..\nYum plugins We can query the repository for packages which put files in the /usr/lib/yum-plugins directory. We have two options for that: using yum with the provides subcommand, or the repoquery utility.\n# yum provides \u0026#34;/usr/lib/yum-plugins/*\u0026#34; Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: ftp.tudelft.nl * extras: archive.cs.uu.nl * updates: archive.cs.uu.nl PackageKit-yum-plugin-0.8.9-11.el7.centos.x86_64 : Tell PackageKit to check for updates when yum exits Repo : base Matched from: Filename : /usr/lib/yum-plugins/refresh-packagekit.py Filename : /usr/lib/yum-plugins/refresh-packagekit.pyo Filename : /usr/lib/yum-plugins/refresh-packagekit.pyc kabi-yum-plugins-1.0-2.el7.centos.noarch : The CentOS Linux kernel ABI yum plugin Repo : base Matched from: Filename : /usr/lib/yum-plugins/kabi.py Filename : /usr/lib/yum-plugins/kabi.pyo Filename : /usr/lib/yum-plugins/kabi.pyc subscription-manager-1.10.14-7.el7.centos.x86_64 : Tools and libraries for subscription and repository management Repo : base Matched from: Filename : /usr/lib/yum-plugins/subscription-manager.pyc Filename : /usr/lib/yum-plugins/subscription-manager.pyo Filename : /usr/lib/yum-plugins/subscription-manager.py Filename : /usr/lib/yum-plugins/product-id.pyc Filename : /usr/lib/yum-plugins/product-id.py Filename : /usr/lib/yum-plugins/product-id.pyo Besides the interesting file paths, it doesn\u0026rsquo;t give much more pointers at this moment. Lets try repoquery:\n# repoquery -f \u0026#34;/usr/lib/yum-plugins/*\u0026#34; | sort | uniq kabi-yum-plugins-0:1.0-2.el7.centos.noarch PackageKit-yum-plugin-0:0.8.9-11.el7.centos.x86_64 subscription-manager-0:1.10.14-7.el7.centos.x86_64 subscription-manager-0:1.10.14-8.el7.centos.x86_64 subscription-manager-0:1.10.14-9.el7.centos.x86_64 yum-langpacks-0:0.4.2-3.el7.noarch yum-plugin-aliases-0:1.1.31-24.el7.noarch yum-plugin-aliases-0:1.1.31-25.el7_0.noarch yum-plugin-auto-update-debug-info-0:1.1.31-24.el7.noarch yum-plugin-auto-update-debug-info-0:1.1.31-25.el7_0.noarch yum-plugin-changelog-0:1.1.31-24.el7.noarch yum-plugin-changelog-0:1.1.31-25.el7_0.noarch yum-plugin-fastestmirror-0:1.1.31-24.el7.noarch yum-plugin-fastestmirror-0:1.1.31-25.el7_0.noarch yum-plugin-filter-data-0:1.1.31-24.el7.noarch yum-plugin-filter-data-0:1.1.31-25.el7_0.noarch yum-plugin-fs-snapshot-0:1.1.31-24.el7.noarch yum-plugin-fs-snapshot-0:1.1.31-25.el7_0.noarch Built-in support Since the security plugin does not show up in any of these listings, we use the discovered file path. Searching in this directory shows the existing yum plugins:\n# find /usr/lib/yum-plugins/ /usr/lib/yum-plugins/ /usr/lib/yum-plugins/fastestmirror.py /usr/lib/yum-plugins/fastestmirror.pyc /usr/lib/yum-plugins/fastestmirror.pyo It is clear only fastestmirror is available. Let\u0026rsquo;s analyze the yum binary.\n# file /usr/bin/yum /usr/bin/yum: Python script, ASCII text executable # grep -i security /usr/bin/yum No hit, so we have to look inside the Python script:\n# cat /usr/bin/yum #!/usr/bin/python import sys try: import yum except ImportError: print \u0026amp;gt;\u0026amp;gt; sys.stderr, \u0026#34;\u0026#34;\u0026#34;\\ There was a problem importing one of the Python modules required to run yum. The error leading to this problem was: %s Please install a package which provides this module, or verify that the module is installed correctly. It\u0026#39;s possible that the above module doesn\u0026#39;t match the current version of Python, which is: %s If you cannot solve this problem yourself, please go to the yum faq at: http://yum.baseurl.org/wiki/Faq \u0026#34;\u0026#34;\u0026#34; % (sys.exc_value, sys.version) sys.exit(1) sys.path.insert(0, \u0026#39;\u0026lt;strong\u0026gt;/usr/share/yum-cli\u0026lt;/strong\u0026gt;\u0026#39;) try: import yummain yummain.user_main(sys.argv[1:], exit_code=True) except KeyboardInterrupt, e: print \u0026amp;gt;\u0026amp;gt; sys.stderr, \u0026#34;\\n\\nExiting on user cancel.\u0026#34; sys.exit(1) By catting the file we can see it includes the /usr/share/yum-cli directory. Grepping through this directory quickly shows one pointer on how to detect if we have security support built-in.\n# grep -r security /usr/share/yum-cli /usr/share/yum-cli/cli.py: self.base.updateinfo_filters[\u0026#39;security\u0026#39;] = opts.security /usr/share/yum-cli/cli.py: group.add_option(\u0026#34;--security\u0026#34;, action=\u0026#34;store_true\u0026#34;, /usr/share/yum-cli/cli.py: help=_(\u0026#34;Include security relevant packages, in updates\u0026#34;)) /usr/share/yum-cli/cli.py: help=_(\u0026#34;Include security relevant packages matching the severity, in updates\u0026#34;)) Binary file /usr/share/yum-cli/cli.pyc matches /usr/share/yum-cli/yumcommands.py: \u0026#39;list-security\u0026#39; : \u0026#39;list\u0026#39;, /usr/share/yum-cli/yumcommands.py: \u0026#39;info-security\u0026#39; : \u0026#39;info\u0026#39;, /usr/share/yum-cli/yumcommands.py: return \u0026#34;[info|list|...] [security|...] [installed|available|all] [pkgs|id]\u0026#34; /usr/share/yum-cli/yumcommands.py: if tn == \u0026#39;security\u0026#39; and notice[\u0026#39;severity\u0026#39;]: /usr/share/yum-cli/yumcommands.py: if tn == \u0026#39;security\u0026#39; and notice[\u0026#39;severity\u0026#39;]: /usr/share/yum-cli/yumcommands.py: if notice[\u0026#39;type\u0026#39;] == \u0026#39;security\u0026#39;: /usr/share/yum-cli/yumcommands.py: for T in (\u0026#39;newpackage\u0026#39;, \u0026#39;security\u0026#39;, \u0026#39;bugfix\u0026#39;, \u0026#39;enhancement\u0026#39;): /usr/share/yum-cli/yumcommands.py: \u0026#39;security\u0026#39; : \u0026#39;Security\u0026#39;, /usr/share/yum-cli/yumcommands.py: for T in (\u0026#39;newpackage\u0026#39;, \u0026#39;security\u0026#39;, \u0026#39;bugfix\u0026#39;, \u0026#39;enhancement\u0026#39;): /usr/share/yum-cli/yumcommands.py: if T == \u0026#39;security\u0026#39; and len(sev_counts) == 1: /usr/share/yum-cli/yumcommands.py: if T == \u0026#39;security\u0026#39; and len(sev_counts) != 1: /usr/share/yum-cli/yumcommands.py: args = (maxsize, sev_counts[sn],sn or \u0026#39;?\u0026#39;, outT[\u0026#39;security\u0026#39;]) /usr/share/yum-cli/yumcommands.py: \u0026#34;sec\u0026#34; : \u0026#34;security\u0026#34;, Binary file /usr/share/yum-cli/yumcommands.pyc matches Great, this provides at least some guidance. For now we use the line with group.add_option to determine that support is built into the yum toolset itself. This enables checking for yum plugins and built-in support.\n","permalink":"https://linux-audit.com/yum-plugins-available-plugins-and-built-in-security-support/","tags":["packages","security","yum"],"title":"Yum plugins: Available plugins and built-in security support"},{"categories":["Auditing","Compliance","PCI DSS compliance"],"contents":" A.1.2.d Verify that viewing of log entries is restricted to the owning entity.\nTo limit exposure to information, PCI DSS requires access of logging to only the entity owning that log file. In other words, we have to search for those entries which can be seen by others.\nSearch related log files By default, most log files on Linux based systems will be stored in /var/log. We can do a quick check for any files which are world readable, by using find.\nfind /var/log -perm -o=r ! -type l\nThis will show all files in /var/log or any subdirectory where the other group has read permissions. We skip any symbolic links, as they will show up otherwise.\nChanging permissions Usually it is easy to restrict log file viewing of these entries by changing file permissions. Depending on the software used, it might be wise to test altering the permissions, restart the process and test if the software can continue to work properly.\nchmod 640 /var/log/nginx/error.log\nAlso tools like logrotate might create new log files with inappropriate permissions. So this control has to be reviewed on a regular basis. It is preferred to use an automated solution to test.\nSome files may need an exception, like /var/log/wtmp. Running the last command will result in a permission denied error.\n$ last last: /var/log/wtmp: Permission denied This information is provided as an addition to the PCI DSS plugin for Lynis.\n","permalink":"https://linux-audit.com/pci-dss-v3-linux-restrict-log-file-viewing-1-2-d/","tags":["compliance","file permissions","log files","logging","pci dss"],"title":"PCI DSS (v3) Linux: Restrict log file viewing (A.1.2.d)"},{"categories":["Crypto","Software","System Administration","Vulnerabilities"],"contents":"What is the Poodle vulnerability ? The \u0026ldquo;Poodle\u0026rdquo; vulnerability is basically an attack on the SSL 3.0 protocol. It was discovered in October 2014. The flaw is in the protocol itself (not implementation), which makes the issue applicable for all products using SSL 3.0. TLS 1.0 and later are considered safe against the attack.\nHow does the attack work? While we won\u0026rsquo;t go into too much depth of encryption and ciphers, we will share some basics. When SSL 3.0 is used in CBC mode, it uses a block cipher. Small blocks of data are being evaluated for further processing, opposed to encryption on bit level.\nPadding During the decryption cycle, the last byte of each block is inspected. It will expect a value between 0 and 7, telling how much padding space was added. Padding is simply a filler. With the attack these reference bytes are removed, which makes it unclear how much padding was added. This results in valuable data being ignored. This could lead to unexpected behavior and forms the basis of putting in other code to abuse a weakness.\nHow to test if I\u0026rsquo;m vulnerable? Most systems have OpenSSL installed. Although this package got a bad attention lately, it is still fine to test for this vulnerability.\necho \u0026#34;GET /\u0026#34; | openssl s_client -ssl3 -connect localhost:443 2\u0026gt; /dev/null | grep \u0026#34;no peer certificate available\u0026#34; \u0026gt; /dev/null || echo \u0026#34;Vulnerable\u0026#34; This will send a normal GET request to the HTTPS server (localhost). It expects to get a \u0026ldquo;no peer certificate available\u0026rdquo;. If not, then that means the connection is accepted (which is bad) and displays the message.\nThis snippet can be used to test if your systems are vulnerable. Make sure the target is alive and running a webserver on port 443, or you get a \u0026ldquo;Vulnerable\u0026rdquo; message as well.\nHow do I solve Poodle? First we have to search for all virtual hosts which have a SSL protocol defined. Each line that does not contain \u0026ldquo;-SSLv3\u0026rdquo; is vulnerable to Poodle.\nSearch for all lines containing a SSL protocol definition.\nApache grep -i -r \u0026quot;SSLProtocol\u0026quot; /etc/apache\nReplace these lines with:\nSSLProtocol all -SSLv2 -SSLv3 This tells Apache to use all protocols, except the weak SSL 2.0 and SSL 3.0 protocols. Do not forget to actually restart Apache on the system.\nNginx Search for all lines containing a SSL protocol definition.\ngrep -r ssl_protocol /etc/nginx Change the found references into:\nssl_protocols TLSv1.2 TLSv1.3; Additional references OpenSSL: Poodle SSLv3 vulnerability ","permalink":"https://linux-audit.com/vulnerabilities/protect-linux-systems-against-sslv3-poodle-vulnerability/","tags":["openssl","ssl","tls","web"],"title":"Protect Linux systems against SSLv3 Poodle vulnerability"},{"categories":["Lynis","System Administration"],"contents":"Tutorial for Lynis installation on Arch Linux\nPacman Arch Linux is getting more popular due to its great community support and the way it is organized. Being a \u0026ldquo;rolling release\u0026rdquo; system, it is continuously up-to-date. Still, you want to make sure your security defenses are equally up-to-date, so that\u0026rsquo;s where Lynis comes in.\nNormally pacman is used for installing new packages. Unfortunately, the lynis package does not show up.\n# pacman -Ss lynis # pacman -Ss rkhunter community/rkhunter 1.4.2-1 Checks machines for the presence of rootkits and other unwanted tools. This is because the package is available in AUR , the Arch User Repository and not yet in the community repository. So we can use Yaourt to install Lynis from there.\nUsing Yaourt and AUR Yaourt installation In case you don\u0026rsquo;t have Yaourt installed, follow the two steps below.\nAdd to /etc/pacman.conf:\n[archlinuxfr]\nSigLevel = Never\nServer = http://repo.archlinux.fr/$arch Now install Yaourt:\npacman -Sy yaourt\nInstallation of Lynis $ yaourt lynis 1 aur/lynis 1.6.2-2 (87) Security and system auditing tool to harden Unix/Linux systems ==\u0026gt; Enter n° of packages to be installed (ex: 1 2 3 or 1-3) ==\u0026gt; ------------------------------------------------------- ==\u0026gt; 1 That\u0026rsquo;s it! Your package has been installed and it ready to use. Make sure to keeping your packages up to date on a regular basis.\nyaourt -Syua\nVoting We love to get Lynis into the community repository, so you can use pacman to install it and keep it up-to-date.\nSo if you like Lynis, log in at AUR and press the vote button.\nWith enough votes the package will be promoted to the mainstream software repositories, simplifying the installation even more.\nAnother option is using the aurvote package.\n$ yaourt aurvote 1 archlinuxfr/aurvote 1.6-1 Tool to vote for favorite AUR packages 2 aur/aurvote 1.6-1 (1239) Tool to vote for favorite AUR packages 3 aur/aurvote-git latest-2 (2) Vote for your favorite AUR packages (development version) ==\u0026gt; Enter n° of packages to be installed (ex: 1 2 3 or 1-3) ==\u0026gt; ------------------------------------------------------- ==\u0026gt; ","permalink":"https://linux-audit.com/lynis/installation-of-lynis-on-arch-linux-systems/","tags":["arch linux","lynis","pacman"],"title":"Installation of Lynis on Arch Linux systems"},{"categories":["System Administration","Vulnerabilities"],"contents":" Shellshock tests for bash in auditing tool Lynis\nShellshock is a serious software weakness, or vulnerability, in Bash. This shell is used on almost all Unix based systems, including Debian and Ubuntu. As it can be used without much effort and remotely exploit systems, it has a maximum vulnerability score according to CVSS .\nUpgrade Bash First update the software repository with apt-get, using the update parameter.\napt update \u0026amp;\u0026amp; apt install -only-upgrade bash\nYour system should now have a newer version of bash. You can check this by using ls -l /bin/bash and see the date of the binary. Additionally, use dpkg -s bash to see package details and the version. Also the changelog of the package will give additional insights regarding the package and what has been done in the latest version.\nShow changelog:\napt changelog bash\nUnattended upgrades To keep up-to-date automatically, use the unattended-upgrades package.\nInstallation Installation is by using apt-get, like you would normally do when using software packages on Debian and Ubuntu.\napt install unattended-upgrades\nConfiguration The only thing which needs to be configured is the configuration file. The package itself includes the following files:\n/etc/pm/sleep.d/10_unattended-upgrades-hibernate\n/etc/logrotate.d/unattended-upgrades\n/etc/init.d/unattended-upgrades\n/etc/apt/apt.conf.d/50unattended-upgrades\nThe last file is the configuration file and you want to check if it includes the security updates.\n// Automatically upgrade packages from these (origin:archive) pairs Unattended-Upgrade::Allowed-Origins { \u0026#34;${distro_id}:${distro_codename}-security\u0026#34;; // \u0026#34;${distro_id}:${distro_codename}-updates\u0026#34;; // \u0026#34;${distro_id}:${distro_codename}-proposed\u0026#34;; // \u0026#34;${distro_id}:${distro_codename}-backports\u0026#34;; }; Within this configuration file, you can decide to do more upgrades unattended, depending on the type of system and your personal preference. After configuration, check if the package has been executed the day after. It will log its activities in /var/log/unattended-upgrades/unattended-upgrades.log. Additional actions are available in the same directory, in separate log files.\n","permalink":"https://linux-audit.com/how-to-solve-shellshock-on-debian-and-ubuntu/","tags":["bash","debian","one-time","shellshock","software","ubuntu"],"title":"How to solve Shellshock on Debian and Ubuntu"},{"categories":["Software"],"contents":"Not everyone has the budget to buy an expensive software suite to do host discovery on the network. Fortunately there are some great open source alternatives. By combining the right tools we can discover hosts and filter the ones we are looking for.\nIn this article we have the goal to determine what systems on our network are running Linux. Of course it is easy to swap out some pieces in the examples to do the same for Windows, Mac OS or BSDs.\nSetting up the toolkit First of all we need to install the proper tools. For our toolkit and this article, these are:\nnmap xmllint For both tools it is better to have one of the latest versions. This is especially true for a better precision of the host detection capabilities in nmap.\nDebian/Ubuntu:\napt install nmap libxml2-utils\nAfter installation check if both commands work, so we are sure they are available.\nScan the network Next step is to scan the network. Since we want to know the operating system, we use the -O option with Nmap. For best results we save the data in a XML file, so we can query that later and extract the data we need.\nnmap -O -oX nmapscan.txt 10.0.1.1/16\nIt might take nmap a while to scan the network. To see the progress press enter and it will show an estimated guess on how far it analyzed the network, discovered hosts and related services.\nParse the data After a while Nmap is finished and we are ready to look at the scan results. With the data already being exported in XML, we can easily parse the data.\nIn our case we want to show only the systems which are most likely running Linux. Since we are interested in the IP addresses for further follow-up, we only list entries which have an actual IPv4 address. So we search the XML file for Host entries, where the Status has a value \u0026ldquo;up\u0026rdquo; for the state field. Additionally we want only the entries within Os, which fall under the OS family of Linux. For these entries we check if there is a IPV4 filled in and then show it on screen.\necho \u0026quot;xpath //host[status[@state='up'] and os[osmatch/osclass[@osfamily='Linux']]]/address[@addrtype='ipv4']/@addr\u0026quot; | xmllint --shell nmapscan.txt\nThe output might look like this:\nFiltering out Linux host from Nmap scan results\nWhile no option is 100% precise, we consider a system to be \u0026ldquo;Linux\u0026rdquo; if the osfamily key is set to this value. Some false positives might show up, like appliances which also run Linux. With the gathered data it is usually easy to determine if we have \u0026ldquo;strange\u0026rdquo; machine, for example when doing a reverse lookup on the IP address.\nFor easier export of the data to a file, use a grep together awk, and we have our list of systems running Linux.\necho \u0026quot;xpath //host[status[@state='up'] and os[osmatch/osclass[@osfamily='Linux']]]/address[@addrtype='ipv4']/@addr\u0026quot; | xmllint --shell nmapscan.txt | grep \u0026quot;content=\u0026quot; | awk -F= '{ print $2 }'\nWith this information we can run a script against this list to check if SSH is accessible, do reverse lookups, or simple store them for later analysis.\n","permalink":"https://linux-audit.com/linux-host-discovery-with-nmap/","tags":["awk","nmap"],"title":"Linux host discovery with Nmap"},{"categories":["Accounting","Auditing","Logging"],"contents":"Logging commands on Linux with Snoopy\nOur customers often want to set-up an audit trail for accounting purposes. When something happens, they want to be able to see what happened, when it did and by whom. Defining an audit trail is also becoming mandatory for compliance, like PCI. One possible solution we cover is using Snoopy, a small library to log executed commands.\nHow it works Snoopy is a wrapper around the execve() function. This is a Linux kernel call which instructs it to execute a command pointed to by a filename. This filename is then logged to syslog, together with any parameters. The related syslog level is authpriv. Usually these events on this level will show up in the file /var/log/auth.log.\nInstalling Snoopy Debian / Ubuntu apt install snoopy\nDuring installation it will ask your permission to add the wrapper to /etc/ld.so.preload, so it can be executed and act as a middle-man.\nIf the library is listed, new commands should be \u0026ldquo;intercepted\u0026rdquo; and logged to your auth.log.\ntail /var/log/auth.log\nThe installation of Snoopy is easy and quick. No further configuration is needed at this point, although you might want to consider to configure remote syslog. This way the log (and audit trail) is stored on an external location\n","permalink":"https://linux-audit.com/creating-audit-trails-logging-commands-linux-snoopy/","tags":["audit","auditing","logging"],"title":"Creating audit trails – Logging commands on Linux with Snoopy"},{"categories":["Vulnerabilities"],"contents":"Bash is one of the most used shells on Unix based systems. The newly discovered \u0026ldquo;shellshock\u0026rdquo; vulnerability affects millions of systems.\nThe weakness abuses an internal check when Bash gets a variable declaration. By defining this variable and putting more \u0026ldquo;stuff\u0026rdquo; (commands) in it, Bash will actually execute those commands as well. Unfortunately this results in several possible ways to exploit it by attackers.\nWebsites One way this vulnerability scan be exploited, is by embedding it in HTTP requests. The extra payload might be the value for a cookie. Some CGI scripts, which reference to Bash as their shell, then will be tricked in executing commands when parsing the value of the cookie. Needless to say, but from there anything is possible, from revealing the contents of files, to implementing a backdoor.\nIf you are hosting websites or use common products (like cPanel), we strongly suggest to look at their forums for the real impact. There is a lot of hype going on on what is vulnerable, but the best source is still your vendor.\nEarly Christmas Apparently this issue was undiscovered for a long time. Only recently the issue was revealed and become quickly a very hot subject. While it is common that issues are detected soon by attackers after a patch has been released, it is less common that the issue is already exploited this quickly. Looks like it is an early Christmas for attackers..\nProtecting against Shellshock To protect yourself against Shellshock, we advice taking the following 3 steps:\nUpdate your software repository and install security updates Check your log files of the last days for traces Audit your system with Lynis Lynis and Shellshock Most security vendors cover this vulnerability as big news and provide additional detection methods in their products. While that is great, companies and individuals should apply security patches already on a regular basis. The addition to a product should be better used as an additional way to check you finished your patching properly.\nPerform manual update Debian/Ubuntu based systems:\napt install --only-upgrade bash\nRed Hat based systems:\nyum update bash\nAutomatic security updates For example Debian based systems can use the \u0026ldquo;unattended-upgrades\u0026rdquo; package to install security related updates automatically. Most other Linux versions have a similar way to split normal and security updates. This enables system administrators to automatically install security patches.\nRegular audits Lynis has already built-in tests to detect vulnerable packages. By running Lynis on a regular basis, it becomes clear that automatic security updates are a good way to stay current.\nAuditing is another detection capability to make sure that these controls are working correctly.\nThe community edition of Lynis has the related tests to check if it can find a \u0026ldquo;package audit tool\u0026rdquo;, which then is used to find possible vulnerable packages. If you have any, they will be displayed at the end (in the report section).\nAm I vulnerable? If you want to test if you are still vulnerable after patching your system, use this code below:\nenv X='() { (a)=\u0026lt;' bash -c \u0026quot;echo echo testing\u0026quot; 2\u0026gt;/dev/null; [[ \u0026quot;$(cat echo 2\u0026gt; /dev/null)\u0026quot; == \u0026quot;testing\u0026quot; ]] \u0026amp;\u0026amp; echo \u0026quot;vulnerable\u0026quot; 2\u0026gt; /dev/null\nThis code will try to set a variable X. It abuses the weakness by creating a file (echo). The script then tries to show it (cat echo). If that succeeds, it will display \u0026ldquo;vulnerable\u0026rdquo; on your screen. In that case your system is not properly patched yet.\nNotes:\nTo avoid warnings on screen, our example redirects errors to /dev/null If you found your system to be vulnerable: Update your Bash package and remove the file \u0026ldquo;echo\u0026rdquo; from your current directory. Or else it will keep saying your system is vulnerable, even after patching. If you get just \u0026ldquo;echo testing\u0026rdquo; on screen, your system is protected. Last week another simple snippet showed up to test:\nvar='() { echo still not fully patched; }' bash -c var\nIf it shows the text, there is still work to do..\nMore information Related CVE\u0026rsquo;s:\nCVE-2014-6271 (up to bash43) CVE-2014-7169 (up to bash43-025) CVE-2014-7186 (up to bash43-026) CVE-2014-7187 (up to bash43-026) CVE-2014-6277 (up to bash43-026) CVE-2014-6278 (up to bash43-026) ","permalink":"https://linux-audit.com/vulnerabilities/protect-shellshock-bash-vulnerability/","tags":["bash","security updates","shellshock","vulnerabilities"],"title":"How to protect yourself against Shellshock Bash vulnerability"},{"categories":["Docker"],"contents":"Things about containers\nSecurity is hot and so is Docker. During the last years Docker has become one of quickest growing container techniques for Linux. While system virtualization continues to grow, the technology allowing flexible containers is growing even faster and starting to compete. In this article we have a look how containers may help you with you security needs and Docker security in particular.\nHow containers work Containers are like \u0026ldquo;chroot\u0026rdquo; on steroids. Where chroot is faking a new directory structure for processes, a container can do this one multiple levels. This includes the file system, network, IPC (inter process communication), users and also process IDs. Docker, being one of the container solutions, is using two areas from the Linux kernel: namespaces and control groups.\nNamespaces Most of the technology behind containers is achieved by using the Linux namespaces technology. By isolating processes, they can\u0026rsquo;t see each other anymore. Within namespaces we have:\nIPC (inter-process communication), like sockets Mounts, changing file systems so a process can only see a part (or only read-only access) Network, including network interface, change routes and specific iptables Process IDs, displaying different IDs User mapping, listing different users to each process UTS (for hostname) Control Groups Next implemented in Docker is cgroups, an abbreviation for control groups. This feature set allows the system to deal with a set of processes regarding their priority and resources. Limiting CPU time, disk IO or maximizing the amount of memory are examples of this. Then there is accounting, which is great for billing your customers or troubleshooting. Last there is the control feature, which allows freezing, restarting and other related command controls for processes.\nSo how does it help? All these features sound great and make sense. By restricting (some) resources systems run smoother. With namespaces we can limit connections only to what is required, increasing our security defenses.\nThere is however a big misconception when it comes to comparing containers and virtual machines. Both technologies look similar on the first glance, but clearly have a big difference: In a virtual machine everything is separated, including the OS, configurations and applications. Containers on the other hand look like a virtual machine when you are inside the container. However, from the host containers just look like normal processes.\nBest practices When working with containers, one should take the same precautions as when dealing with a normal host.\nRoot Processes should not be executed under the root context. There is no excuse of doing so. Of course with the normal exception for master process, which then drops privileges. Just use a non-privileged user ID, like www-data.\nImages When using Docker images, don\u0026rsquo;t leak their unique IDs as they may give others access to your (company) information or intellectual property.\nConclusion\nContainers are hot and you should definitely have a look at them to see if they fit your needs. However, they are not a replacement for security. Proper usage of containers will gain you availability, flexibility and when doing it right, a nice increase in security as well. So our advice: use them wisely.\n","permalink":"https://linux-audit.com/docker-security-how-containers-not-help-you/","tags":["cgroups","containers","docker","ipc","namespaces"],"title":"Docker Security: How Containers (Not) Help You"},{"categories":["Auditing"],"contents":"Every system needs some level of protection. Still, many people simply forget to do it, or can not find the time to properly do it. To be as efficient and effective as possible, let\u0026rsquo;s take at a structured way for security scanning your Linux machines.\nThe 5 dummy steps are:\n1. Focus on risk Like not every company is a bank, our systems are not all part of a top secret mission. We have to look at the risks involved. By focusing on the threats to the system, and the possible vulnerabilities, we can eliminate a lot of work.\n2. Go for the quick wins first We all want to make progress. While it might sound great to have the latest IDS technology implemented, it might also cost a lot of time and money. So instead, get some quick wins implemented first, then go for the bigger impact changes.\n3. Monitor your changes After implementing changes and go to the next one, it is easy to have your improvements being undone by other colleagues, or yourself.. To counter these newly weaknesses in your defenses, measure your security levels and implement monitoring. If something changes which was unintended, get an alert out.\n4. Have others test your systems Sure, you are smart and you know it all. But might there a possibility you overlooked something? Sure.. Get a friendly colleague or your Linux buddy and have them look at your security efforts. You might pickup some new things on the way!\n5. Document The hardest thing in security is knowing what you have done, why and when. Properly documenting it, will save you a lot of time afterwards. Use a great spreadsheet, your change management tool, or even configuration files. Whatever you pick, document it! Add the change, the date, the reasoning and the author of the change. Your colleagues will be thankful for you in the future.\n","permalink":"https://linux-audit.com/linux-security-scanning-for-dummies/","tags":["audit","linux"],"title":"Linux Security Scanning for Dummies"},{"categories":["Hardening","Software"],"contents":"What is this BEAST? BEAST, or \u0026ldquo;Browser Exploit Against SSL/TLS\u0026rdquo; is an attack against the cipher block chaining (CBC) method used with SSL/TLS. The weakness was discovered in 2002, but finally proven in 2011 by security researchers Thai Duong and Juliano Rizzo. With real proof of concept code, they showed it was no longer a theoretical attack.\nTo successfully perform the BEAST attack, there are some conditions which needs to be met:\nVulnerable version of SSL must be used using a block cipher (CBC in particular) JavaScript or a Java applet injection. Should be in the same origin of the web site Data sniffing of the network connection must be possible. Protecting against BEAST attack While it is interesting how the attack work, it is even easier to start protecting your systems.\nTo guard against the attack, we have to define what ciphers we allow. Secondly, we have to set our preference of the ciphers to be determined by the server (instead of the client). Next, we define what protocols we want to use, resulting in older SSL versions to be disallowed.\nssl_ciphers RC4:HIGH:!aNULL:!MD5;\nssl_prefer_server_ciphers on;\nssl_protocols TLSv1 TLSv1.1 TLSv1.2;\nRegarding the ciphers, we can be more specific that list above. We list specifically what ciphers we want to allow by defining the full list:\nssl_ciphers \u0026ldquo;EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS\u0026rdquo;;\nFor a secure list of ciphers, have a look the Mozilla server configuration guide .\nAfter making the changes, reload Nginx. Now it\u0026rsquo;s time to test your Nginx configuration. The SSL Labs from Qualys is a good way to test the configuration.\n","permalink":"https://linux-audit.com/web/protect-against-the-beast-attack-in-nginx/","tags":["hardening","nginx","ssl","tls"],"title":"Protect against the BEAST attack in Nginx"},{"categories":["File Integrity","Monitoring"],"contents":"The most important areas with information security are preventing some events from occurring and detecting it if something still happens. Unfortunately most companies forget to put enough effort in detection unauthorized activities.\nIn this article we have a special look at monitoring your file system, to detect changes to your critical system files and their configuration files.\nMethod 1: File Integrity tools The first method is monitoring file changes with the help of specific tools. These tools usually created \u0026ldquo;hashes\u0026rdquo; of files and store them in a database. Hashes are small cryptographic signatures of a file. Similar techniques and terms are a checksum or parity information.\nWith the help of these techniques, the file integrity can be determined. It works by comparing a newly created hash of a file, with the one stored earlier. If there is a mismatch, the tools will alert the system administrator.\nCommon tools: AIDE , Samhain, Tripwire\nMethod 2: Use Linux Audit framework Another method to detect file system changes, is monitoring these files via the Linux Audit Framework. Any file changed while being monitored, will fire an event and log it in an audit log. The Linux Audit framework is a very versatile solution to monitor changes to system files, but it can do more!\nBesides monitoring files, it can also check for specific system calls. For example a system call to change the time of the machine, is clock_settime. While a related event as changing the time might be less risky than changing the /etc/passwd file, you want to make log an event for purposes of account and forensics.\nPrevention VS Detection While preventing issues is good, detecting them might be more valuable. This is especially true when considering you can\u0026rsquo;t protect against 100% of the threats. Opposed to only try preventing issues, it\u0026rsquo;s actually better to know something happened and then act and improve upon it.\nIf you care about the security of your system, use a combination of both methods. First implement prevention measures to counter most attacks. Secondly implement measures to detect the proper working (and possible failures) of your prevention measures. The usage of file integrity tools and Linux auditing together, will span a big area to cover the detection of intrusions or unauthorized activities.\n","permalink":"https://linux-audit.com/monitor-for-file-system-changes-on-linux/","tags":["file integrity","monitoring"],"title":"Monitor for File System Changes on Linux"},{"categories":["Auditing","Logging"],"contents":"Log files are the precious collection of system events. Still many people don\u0026rsquo;t use them, until it is really needed. Let\u0026rsquo;s go from the reactive use of log files to a proactive stance.\nThe Logging Dilemma Capturing events helps in troubleshooting. By defining what events are ignored and which ones are logged, we get a quick overview on the status of a system. The dilemma is usually in how much logging is enough to get a fair picture and when is it too much. Too less and you lose valuable information, too much and information is hard to find.\nWhat to capture Depending on the goal of the machine, it helps to create a small matrix with the main services running on the system. Then define what the bare minimum information is required per service to be logged. For example when running nginx, you might want to log all major events of nginx itself (crashes, configuration errors), while limiting debug information. Additionally you want to define per website a log file to capture access requests for data analysis purposes. One log file for the errors (per website) for troubleshooting and optimizing the particular website.\nLooking for the needle Log files are usually simple text based files. With your favorite text manipulation tools (e.g. grep, awk, sed) you can quickly search through log files and find the events you are looking for.\nLog files and Automation The handling of log files can be automated fairly easy. Yet there are some options to consider:\nLocation:\nKeep all log files local Store local and send to remote syslog host Log events to SIEM solution Data handling:\nStore as-is Store and forward data with filtering With rsyslog you can directly filter events and store them in separated files. This makes reviewing logs much easier. Tools like logcheck, swatch and logwatch may be helpful as well for monitoring logs and filtering out \u0026ldquo;noise\u0026rdquo;.\n","permalink":"https://linux-audit.com/linux-security-reviewing-log-files/","tags":["audit","log files","rsyslog","syslog"],"title":"Linux security: Reviewing log files"},{"categories":["Shell scripting"],"contents":"Our security auditing tool Lynis is a toolkit consisting of several shell scripts. Many users of the software actually never realized it was written as a bunch of shells scripts. Now that the secret is out, it is time to learn why we used shell scripting. Here are 5 reasons!\n1. Shell scripting is powerful Yes, people asked us why our tool Lynis was not written in Perl, Python, Ruby, C++ etc.. But honestly, why would we? It is portable, no compilation needed and supports almost every single Unix based system there is. Combine it with the common available tools like awk, grep and sed and you have a great foundation.\n2. Shell scripts don\u0026rsquo;t have to be boring Most shell scripts are boring. They don\u0026rsquo;t show anything, nor have any coloring. What a shame.. Lynis uses a color scheme, has an update check, intercepts interruption (e.g. CTRL-C) and shows warnings if it was not terminated properly last time. There is so much possible!\n3. Repeat, repeat, repeat Why repeat the same statements in your shell scripts each time? Build a powerful set of functions and include that in your existing and new shell scripts. Don\u0026rsquo;t use \u0026ldquo;echo\u0026rdquo; while you also can call your own function Display. By using your own function you can now determine on-the-fly if this output needs to go to the screen, or maybe be suppressed, or get logged as well. Let your script do the smart thinking and redirecting.\n4. Readability Whatever programming language you use, it can be quickly become a burden to understand what happens and why. Remember when reading your code after a few years (or just a few weeks..)? Yes, you need to use proper commenting, but that\u0026rsquo;s also area of science itself.\nWith shell scripts the chance of making things really unreadable is much lower. Sure you can use exotic features of the shell, which others don\u0026rsquo;t understand. But if you honestly care about your scripts, you can make it workable and at the same time keep it readable. The beauty of simplicity!\n5. Always available Being able to do real programming is great. But if you are a system administrator, you might not enjoy programming at all. A shell script is different.\nShell scripting can always be used, on each system you encounter. It makes your life easier by automating repeated steps. No fiddling with code, setting crazy memory pointers or discovering why used a wrong file descriptor. Just paste your favorite commands in a file, make it executable and run it proudly. Easy to learn, and, well also easy to master.\nOh, although we love the power of shell scripting, there is no doubt that other alternatives may be a better fit for your work. We definitely don\u0026rsquo;t want to \u0026ldquo;bash\u0026rdquo; your favorite programming language ;)\n","permalink":"https://linux-audit.com/5-things-didnt-know-shell-scripting/","tags":["automation","one-time","linux","programming","shell script"],"title":"5 things you didn’t know about shell scripting"},{"categories":["Hardening","Software","Web"],"contents":"HTTP Strict Transport Security (or HSTS) is a security capability to force web clients using HTTPS. The idea behind HSTS is that clients which always should communicate as safely as possible. At achieve this, the web server and web browser will prefer the HTTPS protocol instead of HTTP.\nBenefits The clear benefit of \u0026ldquo;forcing\u0026rdquo; a client to use HTTPS directly, is decreasing the risk of sharing any sensitive information via a protocol which can be snooped upon. Additionally it improves the performance by eliminating one redirect response (301/302). Another benefit is to force using a secure connection and deny a client if this can not be guaranteed (e.g. expired or self-signed certificate).\nHTTPS configured with HTST, HPKP and forward secrecy.\nConfiguration Configure HSTS on Apache Load the headers and mod_rewrite module (just to be sure)\n# Load modules (or use the IfModule) LoadModule headers_module modules/mod_headers.so LoadModule rewrite_module modules/mod_rewrite.so Rewrite HTTP connections and redirect them to HTTPS:\n# Redirect HTTP connections to HTTPS \u0026lt;IfModule mod_rewrite.c\u0026gt; RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L] \u0026lt;/IfModule\u0026gt; Now configure the virtual host:\n\u0026lt;VirtualHost 192.168.1.1:443\u0026gt; Header always set Strict-Transport-Security \u0026#34;max-age=31536000; includeSubDomains\u0026#34; \u0026lt;/VirtualHost\u0026gt; Configure HSTS on Nginx To use HSTS on Nginx, use the add_header directive in the configuration. Then tell clients to use HSTS with a specific age.\nadd_header Strict-Transport-Security \u0026#34;max-age=31536000; includeSubDomains; preload\u0026#34; always; max-age: expire time includeSubDomains: also apply this protection on our subdomains preload (optional): preload list in some browsers, requires a max-age with a minimum value of 31536000 The always statement at the end of the add_header setting ensures that the header is always inserted in the response headers.\nAdjust the related virtual hosts to perform a redirect (301) to the secured version of the website:\nhttp { server { listen 80; server_name example.com return 301 https://$server_name$request_uri; } server { listen 443; server_name example.com add_header Strict-Transport-Security \u0026#34;max-age=31536000; includeSubDomains; preload\u0026#34; always; } } Important notes The HSTS header should only be sent over a secured channel, therefore HTTP responses should not include them.\nWithin the headers, the max-age defines what period the site is willing to accept HTTPS-only (31536000 in the examples are 12 months). Usually, the amount of time is less important. This is because the trend is to keep using HTTPS for privacy and data protection anyways.\nAdditionally, make sure the domain itself is also properly configured for HSTS. This reduces attacks on the underlying subdomain names.\n","permalink":"https://linux-audit.com/web/configure-hsts-http-strict-transport-security-apache-nginx/","tags":["apache","hpkp","hsts","nginx","web browser","web server"],"title":"Configure HSTS (HTTP Strict Transport Security) for Apache and Nginx"},{"categories":["Linux"],"contents":"Administrators of Debian-based systems know they have to reboot their systems, just like any other Linux distribution. However, why is the reboot needed? Could we monitor for which systems need an actual reboot?\nRequired restart required? This Ubuntu system needs a restart\nRequired reboot Software can contain issues, which we call bugs. Most bugs are just annoying if you encounter them and can be fixed by upgrading to a newer version of the software. Other bugs are special in the way that they may leak sensitive data or allow unauthorized access to the software or system. These type of bugs are called vulnerabilities.\nTracking which servers need a required reboot is important to properly solve vulnerabilities. Installing a software update is a first good step, but sometimes more is needed. For most software the restart of the related software processes is sufficient. For others, a system reboot is needed. Especially with weaknesses in the kernel or related to global components (e.g. Glibc, OpenSSL).\nFortunately, we can check if a reboot is needed. If the file /var/run/reboot-required.pkgs exists, then one or more processes require a full reboot. The file won\u0026rsquo;t show process names, but the related packages.\n# cat /var/run/reboot-required.pkgs libssl1.0.0 In this example we see the file exists and contains an update to the SSL library used by the Linux kernel. Since not all libraries can be reloaded that easily, the system has a reboot required.\nThe /var/run/reboot-required.pkgs file\nBoth files will be automatically deleted by the system after a reboot.\nAutomation Most system administrators love to automate everything. This process of monitoring can be automated as well. Tools like Lynis will check for the presence of /var/run/reboot-required.pkgs and list which packages are inside the file. Since a kernel reboot is important, it will create a warning event and display this in the report. Monitoring which servers now need a reboot has become much easier.\nAnother possibility is to add this check to your network and system monitoring tools as well. With some basic scripting, the check can be implemented easily.\nAutomatic reboot For people who really love the next level of automation (and love some risk), they could automatically schedule a reboot event. If the file has been found, create a one-time event to reboot the system with your favorite configuration management tool (like CFEngine, Chef or Puppet).\n","permalink":"https://linux-audit.com/check-required-reboot-on-debian-ubuntu-systems/","tags":["debian","linux","reboot","security updates","ubuntu"],"title":"Check for a required reboot on Debian and Ubuntu systems"},{"categories":["Auditing","Programming"],"contents":"In this article we have a look at the privileges of Linux daemons and dropping privileges in particular. The samples provided are in C.\nWhy drop privileges? Some daemons need root permissions to start. This happens for example when a daemon wants to bind to a low port (\u0026lt;1024). However running network based daemons with root permissions is considered to be a serious risk. In case of compromise of the process, an attacker has full access to the system. This is why software like nginx starts with a master process and forks non-privileged child processes. These child processes (or workers), run under the context of non-privileged account like www-data.\nroot 2034 1 0 Jun10 ? 00:00:00 nginx: master process /usr/sbin/nginx\nwww-data 2036 2034 0 Jun10 ? 00:00:35 nginx: worker process\nwww-data 2037 2034 0 Jun10 ? 00:00:36 nginx: worker process\nwww-data 2038 2034 0 Jun10 ? 00:00:33 nginx: worker process\nwww-data 2039 2034 0 Jun10 ? 00:00:37 nginx: worker process\nHow to drop privileges First the program needs to check its current user ID. If it is zero, the equal of root, then it should drop both the user ID and group ID. This can be done with the setuid and setgid functions.\nif (getuid() == 0) { /* process is running as root, drop privileges */ if (setgid(groupid) != 0) fatal(\u0026quot;setgid: Unable to drop group privileges: %s\u0026quot;, strerror(errno)); if (setuid(userid) != 0) fatal(\u0026quot;setuid: Unable to drop user privileges: %S\u0026quot;, strerror(errno)); } Once a process has switched to a non-privileged user, it should not be able to regain root permissions. This can be tested with the following snippet:\nif (setuid(0) != -1)\nfatal(\u0026ldquo;ERROR: setuid back to zero succeeded, quitting as this is a security risk\u0026rdquo;);\nIf it succeeds for what reason, the program should terminate.\nAdditional groups The root account may also be part of supplementary groups, besides the usually root or wheel group. With the help of the function initgroups, any of the supplementary groups of the root user can be dropped. This way the process can\u0026rsquo;t access any data by accident.\nCurrent work directory When starting a process, the current work directory might still be something owned by the root user. For safety, the chdir function can be used to move to another safe work directory. Usually this will be the home directory or data directory.\nCapabilities Another way to provide only limited privileges to a binary, is giving them capabilities. This is implemented in the Linux kernel and splits all different \u0026ldquo;roles\u0026rdquo; the root user can have. Examples include this earlier mentioned binding to a low port, open network sockets or loading a kernel module. For daemons it is not wise to run fully under the root context, but you might want to use the capability to open up a port.\nFor more information about how capabilities work, see our Linux capabilities 101 post. There is also a Linux capabilities overview that shows all available capabilities and their purpose.\nConclusion Safe programming takes a lot of effort, but helps not in introducing weaknesses into software, systems and our valuable data. After all it is this same data which needs to be protected, hence every effort is another useful step in achieving this important goal.\n","permalink":"https://linux-audit.com/how-and-why-linux-daemons-drop-privileges/","tags":["auditing","capabilities","kernel","programming"],"title":"How and why Linux daemons drop privileges"},{"categories":["Linux"],"contents":"Information security is possibly one of the hardest subjects in IT. Doing too less and you risk of security breaches. Doing too much will restrict the core businesses of your organization. With a proper security program, implementing Linux security can be greatly simplified. By having a structured approach, the strength of the defenses will increase, while risks decrease. In this article, we have a look at how to properly prepare security projects and changes. This way we can increase the chance to succeed with the security program. This article is a combination of processes and technical aspects, so it will be focused on both project managers and system administrators.\nChange Management Almost everything is changing continuously. This is especially true in IT environments. Companies using ITIL have a related process in place, named Change Management . Even if there is no formal process, changes in organizations should be properly prepared and executed. Sure, things can be changed without any formal process. The risk is so-called configuration drift, where every system is (completely) different.\nBy focusing on several key elements, we can improve the change quality and increase the succeed ratio tremendously.\nThere are 4 key elements to achieve a change.\nNeed for change Vision Resources for improvement Plan and first steps 1. Need for change While it might be obvious for some people that something have to be changed, it is often unclear why the change has to occur. Other people might have to be sold on the idea of the change. Only when having all key people on board, chances to succeed are much higher.\nAdditionally the need for change consists of some pressure. For example regarding time. There should be some sense of urgency, or else people won\u0026rsquo;t start moving.\nReasons for implementing security activities like system hardening or regular security audits, might be for compliance.\nDefine the need to change with lessons from the past, like:\nAny intrusion / break-in Lack of trust by customers Requests from customers for compliance 2. Vision Before trying to change something, get a clear image on what the outcome should be. Vision is needed to get closer to this outcome. Usually it consists of a set of insights, knowledge and decision making. Know your market, environment, company and people.\nIn information security, we also should have a clear vision in why we are here and what we try to achieve. When implementing Linux security in your environment, especially if it currently is based on ad hoc activities, take a few steps back first. Define the outcome, the possible constraints and the key people involved. Write down who is opposite of such change and who might be more than an enabler of the security program.\nFor the Linux security program, make sure to know at least these areas:\nUsed Linux/Unix versions (e.g. Red Hat, Solaris, mixed) Available budget Available resources Deadlines 3. Resources Nothing can be changed without resources. This includes the appropriate tooling for example. Sometimes existing tools might be used, or build upon toolkits and processes.\nBesides tooling, there is people and knowledge. Does the company have enough in-house knowledge, or should an external consultant assist? Even if you have the knowledge, are these people available for side projects?\nThen there is time, deadlines and the related pressure. Make sure there is enough time, or at least enough people to help completing tasks. Time killers include company politics, badly prepared meetings and distractions. Especially people working on projects to enable the change to happen, should not be distracted with (too much) operational tasks.\nRequired information:\nWhich people are active in project Who has a need to know Are there are any deadlines Compliance applicable (SOx, PCI etc) 4. Plan While actual change is especially visible in the execution, it is the planning which makes the transition much smoother. Know possible threats, include what people need extra convincing, or what communication needs to happen.\nTry to have a detailed plan upfront, so it is clear what needs to be done at what stage. Including communication, meetings and approvals (e.g. a moment for a go / no-go).\nTools to use:\nTemplates Progress sheets Change Management (ITIL) Spreadsheet and databases (e.g. CMDB) The outcome of using the right tools and the invested preparation, should be a clear plan with all activities includes. When it is time to start working on the first activities, the plan should be guidance for involved people.\nDealing with exceptions\nMost projects will have sooner or later an exceptional event. Simple events like missing a deadline, due to a third party not fulfilling its promises for delivery. Other exceptions may include technical difficulties during implementation. With a clear plan, the amount of bad surprises should be limited. However if people know how they can report any inconsistencies, project risks can be decreased. Anything reported exceptions can be handled on a case-by-case basis. Determine upfront who has authority to collect exceptions and who is authorized take action or make a decision.\nProgram or Projects? Implementing security on Linux based systems can be a very time-consuming project. Therefore, it might be better to embed it into a program, with many smaller projects.\nEach project will then have a clear scope, time constraints, and dedicated resources assigned. The big benefit is that change is directly visible, as now projects might just take a few hours (instead of days or weeks).\nCore (Linux kernel) With Linux being the core of the operating itself, it makes it the right location to start with. Determine what Linux kernel versions are being used within your environment. Determine what actions already have been taken to secure the kernel and the related components. This might include the way data is stored on the system (what file system is used?). What options are used to fortify the core of the operating system.\nDetermine the version of the used Linux kernels might give a great insight in software patch management. If old kernels are being used, chances are high that the system is vulnerable to attacks via the network, or locally. Usually exploiting tools being quickly available to abuse these weaknesses. This makes it important to start protecting the system from the inside out and patch management is one first step.\nSoftware There are two fundamental things when securing a system when it comes to software:\nUse of what software components Software and security patch management While this document is not meant as a system hardening guide, the first piece is important to keep systems clean. Do not install tools which are not needed. Besides the additional space, it might give access to data in unexpected ways. In the worst case you might be even running an unused but vulnerable application!\nRegarding software patch management you should define a clear plan on how to embed proper software and security patching. How are you going to deal with software updates and how with security updates in particular. Do they follow the same schedule, or are security patches given priority to limit the exposure of weaknesses? Depending on your organization and security policy, this might totally depend.\nNetworking From a networking level things have become more interesting over the last years. Every system nowadays is interconnected, up to our mobile phones. Protecting the network stack is therefore an important part in securing systems.\nOne of the things for a security program could be the implementation of a firewall. Even if you already have a network firewall, the presence of a local firewall might have benefits. This is especially the case when having systems of multiple customers in the same VLANs, or accessible via the internet.\nUsers Last, but definitely not least, is dealing with users. Not the people itself, but dealing with the AAA part of it:\nAuthentication How are users connecting to the system and what kind of authentication controls might they use (password, smart card, token). Closely related are the implementation of password controls, defining the security and strength of these controls.\nAuthorization After a user is successfully authenticated, we should know what each user is supposed to do on a system. This involves in determining what a user could access, in what groups it should be in and the related file system permissions. One of a common type of authorization control is the usage of sudo. If a \u0026ldquo;normal\u0026rdquo; user needs temporary privileged permissions, you do not want to hand them the root password. Properly implementing sudo gives users the possibility to do their work, while keeping the root password secret.\nAccounting For controlling our security, we should know upfront what a user is supposed to do. However, you might want to monitor behavior on the system for security reasons as well. Other reasons might be troubleshooting or debugging. In any case, tracking access and executed commands might be useful.\nSafeguarding of Changes Changing an environment, including Linux systems might be easy. However we should keep in mind the goal of the program and the related projects. Each change should be properly documented, approved and executed.\nAfter each change has been done, documentation should be updated, so operational staff is aware of the change and use the right work instructions. Additionally, the system and change should be protected against other changes (e.g. undoing previous work). By properly monitoring the recently made change, we can control the improvements and make sure they stay in place.\nWith all tips in this document given, good luck with your security program! If you need additional guidance or tips for your program, contact us via the About page.\n","permalink":"https://linux-audit.com/security-program-implementing-linux-security/","tags":["linux","linux security"],"title":"Security Program: Implementing Linux Security"},{"categories":["Auditing"],"contents":"Technical audits or vulnerability scans will reveal a lot of findings. They can be overwhelmed and forcing the reviewer to freeze, not knowing where to start. To overcome this issue, we should prioritize the findings and determine the consequences of each finding for our company.\nWhile an open directory listing on a web server might in one situation not be preferred, it would make sense for others. It is the context which makes a finding \u0026ldquo;serious\u0026rdquo; or completely harmless.\nFocus areas By looking at several aspects, we can simplify and prioritize audit findings. Three common focus areas are Importance (or impact), Urgency and Effort. Depending on the time of audit, these three areas may have the same weight. In most situations however, there is a clear preference for one or more focus areas. When delivering a project, deadlines may be more important, resulting in a higher weight for urgency and also for effort. The latter may give preference to another activity, because it can be a \u0026ldquo;quick win\u0026rdquo; for example.\nImportance or Impact Some activities will have a great benefit to the business, like the trust it provides to customers, cost savings, or the convenience of the work for employees. By determining the benefits of dealing with a particular finding, can put all activities in perspective. This makes it easier to compare them and prioritize them. Usually this focus area is the most beneficial to a company.\nNote: In a normal situation, give this area the highest weight.\nUrgency While some activities might have a high impact, the timing might be less optimal. For example when another activity needs to be completed first, or simply because there is no pressure to change. People tend to change things only if there is a sense of urgency, which is also true when one has to prioritize audit findings.\nUrgency is usually second when rating the weight of each of the focus areas. The higher the urgency, the more pressure there is externally and internally to get a particular item solved.\nEffort Regarding quick wins, it is the effort rating which determines how well suited it is. For most activities however, it quickly becomes clear there is more work involved. By properly determining what amount of effort is needed (in man hours, time), the quick wins will raise to the top. If something is important, there is a sense of urgency and it can be quickly performed, this activity should be done first.\nUsually effort is rated lower than impact and urgency, as effort and benefit usually a synchronous. Something that can be done quickly, has usually not much impact for the business. Some real big changes which provide new options for the business, will take longer. Consider also getting people aboard on the idea, or why a specific finding is really that important. Convincing others is also a measure of effort.\nRating To get a prioritized list, give each finding a score (e.g. 1-5). Then apply the weight to each item and count the scores. Since low effort is good, we should turn around the score rating for this one (score = 6-value). If someone fills in a 5 (which means a lot of effort), it will result in a score of 1. Optionally is to name the item differently, however be careful not to use negative connotations.\nThe scoring can be done easily with a spreadsheet program. Some solutions have their own way of calculation these factors, to save you the time to do it manually.\n","permalink":"https://linux-audit.com/security-audits-prioritize-audit-findings/","tags":["auditing"],"title":"Security Audits – How to Prioritize Audit Findings"},{"categories":["Auditing","Compliance","Hardening"],"contents":"Quality is an interesting word. It describes, well, the quality of something. Quality is just another word for how well can you repeat something. The goal is to get each time exactly the same result. Whenever it\u0026rsquo;s a physical product, or rolling out a new Linux system, you want great quality. One method to increase quality is using checklists. However we strongly advice against using Linux hardening checklists..\nBut checklists are good, right? People forget to do things, which is the reason checklists were invented. By forcing yourself to check individual items on a checklist, the quality of the work is greatly improved. Checklists would be very useful in IT as well. Surprisingly, many IT departments still don\u0026rsquo;t use them.\nWhile we would promote checklists in IT, they are better to be used during administrative tasks. For example when on-boarding a new employee. This includes providing a desk, phone, company handbook, account creation and a personal badge. However for your Linux servers we suggest to use automation.\nAutomation part 1: Configuration Management Use configuration management tools like Chef, CFEngine or Puppet. This enables you to quickly roll-out new systems and putting the basic premises in place. No longer check if a system is in the CMDB (Configuration Management Database), but make it mandatory before a system can be rolled out at all. Any exception (e.g. a manually created virtual machine) should be discovered by scanning the network.\nAutomation part 2: Auditing Never trust on automation tools alone. Perform regular auditing, with yes, automated tools. Additionally perform manual tests. This ensures you that both your automated tools and your control tools, are doing what they are supposed to do.\nAuditing comes in many forms and many can be applied in your environment as well.\nExamples:\nScanning for rogue WiFi access points Testing security defenses Check CMDB data with the results from network management tools Perform vulnerability scanning Test time synchronization By performing regular tests, outliers and exceptions can be greatly reduced. The big issue is that people usually wait too long, until it goes wrong. Then monitoring is reconfigured, until the next issue occurs. IT system administrators should be wearing the hat of the IT auditor more often: question everyone and everything.\nSome questions to ask yourself (and your colleagues):\nWhy is this working this way? Does it really work so and how do you know? Can you show the proof? Conclusion Checklists are a fine tool to improve quality. But with focus on automation we can achieve much more than filling in some administrative forms. Apply configuration management tools and auditing to improve quality and keep checklists for less technical areas. And.. become an IT auditor yourself. Start challenging things, including your own work!\n","permalink":"https://linux-audit.com/do-not-use-linux-hardening-checklists-for-your-servers/","tags":["hardening","linux","system hardening"],"title":"Do NOT use Linux hardening checklists for your servers"},{"categories":["Auditing"],"contents":"How to audit AIX Unix systems with Lynis Each system is as strong as its weakest link. Also for systems running on AIX this rule applies. Therefore a regular audit can help finding the weakest links. Next step is then the fortification of these weakened areas and implementing system hardening measures.\nWhat to audit? There is a lot to look for when auditing a system running AIX. Let\u0026rsquo;s have a look at the most important areas.\nFile systems Monitor alterations to critical systems files. Configurations usually should be similar and properly controlled. Unauthorized file changes are definitely not something you want. Implement tight file permissions and only provide access to users which really need access. Protect data directories which contain sensitive data.\nRelated commands: lsfs, mount\nKernel The core component of each operating system is the kernel. Using safe values and parameters will protect the system from crashing. Proper tuning needs some careful attention though. Consult related documentation when adjusting the kernel, to make sure that the kernel properly deals with network traffic and the right security measures are enabled. Especially with kernel hardening the focus should be correct. Does the system handle a lot of sensitive data? Go for full protection. Is it just action as a gateway and interfacing with users, go for the optimal mix of performance and security.\nRelated commands: no, smtctl\nLogging Monitor for unexpected events (software crashes) to detect weak areas on the system. Also detect common events like login failures and have them logged. They can be also linked with an existing SIEM (security information and event management) solution, or forwarded to the security officer.\nRelated commands: alog, errpt\nMemory and swap Determine memory usage and make sure no processes are hogging up memory in an unexpected way.\nRelated commands: lsps\nNetwork Implement a firewall to limit traffic to what is needed for properly functioning and its business goal. Also check for proper tuning of the system, so it is optimized to deal with the number of users it is facing.\nRelated commands: ifconfig lsattr, netstat, route\nPatch management Software is one of the biggest areas on a system where vulnerabilities can exist. Proper software patch management helps with solving any weaknesses found in software. If there is one area to pay attention to and put some time into it, then it is software management and patching.\nRelated commands: lslpp, oslevel\nHow to audit? We already blogged a few times on what to look for when auditing Unix systems. While we definitely suggest reading other posts, we want to simplify the lives of others. This is where Lynis comes into play. Lynis is an auditing tool for Unix based systems. It runs on almost all Unix platforms and performs an in-depth audit in a few minutes.\nIf you want a quick idea on what areas to improve on your systems, give Lynis a try. It\u0026rsquo;s open source and free to use. With a big community of users, the software is very popular and widespread. The findings showing up can be a great point to start with your system hardening efforts!\n","permalink":"https://linux-audit.com/audit-aix-unix-lynis/","tags":["audit"],"title":"How to audit AIX Unix systems with Lynis"},{"categories":["Auditing","Lynis"],"contents":"After finishing an audit with Lynis, the screen is usually filled with a lot of suggestions. Most users don\u0026rsquo;t know where to start with hardening and how to deal with these Lynis suggestions in particular. We provide you some tips!\nBefore we start, we strongly suggest to use the latest version of Lynis. If you are using an outdated version from the software repositories, the output could be slightly different.\nThe latest version can be downloaded on the downloads page.\nStep 1: Follow the link After each warning or suggestion a link is displayed, which is related to the security control. The website contains more information regarding this control, to prevent the screen filled up with long pieces of text. This text will give an initial idea on what could be improved.\nStep 2: Check the log During the Lynis run, it will collect a lot of additional information. This information can be considered as debug information and is very useful after the scan process. It includes information from the start of the program, OS and binary file detection and the outcomes of each individual test.\nTo quickly determine what has been discovered during a particular test, open the log file with the less command and perform a search for the related control.\nless /var/log/lynis.log\nStep 3: Check the source The big benefit of using open source software components, is the ability to look in the source code. Normally this isn\u0026rsquo;t easy for novice people, as you require some programming knowledge to understand the logic. Fortunately Lynis is written in shell script and the logic is easy to understand.\nWhen looking why some Lynis suggestions showed up, go to the include directory. Perform a grep to check what files is performing a particular test.\n# cd include # grep FILE-1234 * The related filename will show up and with less (or your favorite text editor) the contents can be reviewed. Usually it will quickly become clear what files were tested and what particular text strings are related.\nNotes While we strongly believe that most people can harden their systems, we still see that most companies and people don\u0026rsquo;t properly perform this hardening. This is why we created an Enterprise version to help simplifying this process.\nIf you have more than 10 systems to manage, we strongly suggest to avoid manual hardening. Automation is the key in getting and keeping your systems secure. Whatever method you use, focus on automation and use software configuration management tools like CFEngine, Chef and Puppet.\n","permalink":"https://linux-audit.com/lynis/how-to-deal-with-lynis-suggestions/","tags":["hardening","lynis"],"title":"How to deal with Lynis suggestions?"},{"categories":["Development","Linux","System Administration"],"contents":"During the last years the role of DevOps evolved. This person could be described as the hybrid: a system administrator with development skills, or the developer which is also infrastructure savvy. With Linux and so many available tooling, it is becoming easier for people to learn both development and managing infrastructures.\nWe are especially interested in Linux security for DevOps and what they can apply.\nAutomation is key Repeating work is not only boring, but also a waste of time. Every step which is repeated, might be a great candidate for automation. With solutions like Puppet, is has become easy to automate installations, software installation and configuration.\nSecurity from the start Whenever possible, tighten up your defenses. For example, roll-out iptables on each machines by default, with a standard strict template. When a particular system has to become a web server, let Puppet open up the related web ports.\nSoftware patching Most system administrators are aware of the difficulties when dealing with many different software versions. Therefore, keep an eye on software upgrades and embed software patch management in your routines. Especially security patches should be evaluated at the moment they are released and put into production when appropriate.\nLog \u0026amp; Event management Automate the collection, parsing and alerting for suspicious events. If you are a puppet master, you want to know exactly what is going on. As highlighted in the first point, automation is key. Manual activities should only be done for the exceptions, outliers and control activities.\nBefriend your CMDB The configuration management database, or CMDB, is the central place where all information about systems are located. Properly interfacing with your CMDB will save a lot of time. Most solutions have a way to interact and gather data, like in XML/JSON format.\nAt the very moment a system is installed, decommissioned or offline, all related components should up-to-date. You don\u0026rsquo;t want to turn on a system, which was just replaced by another one.\nDon\u0026rsquo;t build your own islands While it might be great to build your own solutions, try to leverage existing solutions within the company. From the previously mentioned CMDB, used software components, documentation tools or developer repositories. Focus on automation and using the core components which already exist.\nPerform audits Besides automation, also include running regular audits. Simply trusting your configuration management solution is not enough. The Plan-Do-Check-Act cycle is the perfect method to apply. Keep on improving in steps and don\u0026rsquo;t forget the \u0026ldquo;Check\u0026rdquo;.\nSecurity audits will give new insights and room for improvement. If you are keen on keeping your system healthy, perform regular checks. This includes for unauthorized users, specific events in your log files.\nDon\u0026rsquo;t be shy to let others audit your environment. This includes IT auditors, a consultant or colleague. You can\u0026rsquo;t know and have it all. Others might challenge you to think about the best possible solution, while you thought you already had it. There is always room for improvement and it is usually your friendly coworkers who can see it. Trust on your knowledge and be open for input.\n","permalink":"https://linux-audit.com/linux-security-for-devops/","tags":["devops","linux"],"title":"Linux Security for DevOps"},{"categories":["Auditing","Software"],"contents":"This was the big question we asked ourselves recently, when reading a few of them. With Linux and other Unix systems being decently hardened by default, would it still make sense to invest a lot of time to harden your system?\nHardening guides Years ago both Windows and Linux were easy targets. A lot of system software was installed by default and these services were targeted often by malicious people and scripts. Then hardening guides came along on how to secure these services and protect systems.\nMinimal installation After hardening guides used to be a normal practice, vendors were forced to deliver at least an option to use a \u0026ldquo;clean\u0026rdquo; installation. That is, an installation with the bare minimum installed. Under Linux these kind of installations were often called \u0026ldquo;minimal\u0026rdquo;, resulting in a quick installation and only those components needed to run the system. Depending on the role, additional software then could be installed.\nDo we still need them? With vendors delivering better hardened system installations, one could argue that the need for hardening guides then dropped as well. Still, we think the need will remain for many more years to come. After all, a system without running software, is similar to a house where no one lives. It is possible, yet not really useful.\nHardening of software While the operating system maybe be better hardened already, many software components are not. Usually they never will be hardened out-of-the-box as, they need to provide functionality. Secure by default is a nice thing, but most people rather prefer something to work than being secure.\nSo in other words, hardening guides will remain useful. The focus will be more on individual software components and less on the operating system. This is also the reason why our auditing tool Lynis does more than just auditing the operating system. It is the combination of a hardened base system, properly configured system components, focus on networking and more. Only if all chains are strong enough, then you can rely on using it for your precious operations.\n","permalink":"https://linux-audit.com/system-hardening/security-hardening-guides-still-useful/","tags":["guide","hardening","software"],"title":"Are security hardening guides still useful?"},{"categories":["Forensics","Intrusion Detection"],"contents":"Malware, or malicious software is also an issue on Linux systems. Let\u0026rsquo;s have a look into this threat and what actions you can take.\nWhat is a rootkit? A rootkit is a set of tools with the goal to hide its presence and to continue providing system access to an attacker. The word rootkit comes from the root user, which is the administrator account on Linux systems and Unix-clones. The kit refers to a toolkit, or a set of tools.\nHiding by manipulation The tools in the rootkit are typically altered binaries that provide an alternative truth. They will display everything a typical command would do, except those parts that are part of the rootkit itself. Some rootkits also provide an additional backdoor. This way the system can be fully patched and still allow the attacker to enter via the hidden entrance.\nRootkits are usually not installed by a system administrator. In fact, the system administrator is typically the victim of such software. He or she is seeing manipulated results when using common system tools like ls and ps.\nPopularity of rootkits Nowadays rootkits are less popular than they were before. This is partially due to some measures were taken in modern Linux kernel versions. The newer versions increase the difficulty to circumvent some areas like using some of the available system calls. As you may expect, the cat-and-mouse game is not over and backdoors remain very popular. Often the attacker doesn\u0026rsquo;t even need full root access to misuse a system for other purposes. Helping in a Distributed Denial of Service (DDoS), sending spam, or act as a hop to attack other systems, to name a few.\nRootkit detection Methods to detect rootkit presence Since rootkits are malicious, they should be detected as soon as possible. There are different ways to detect them, each with different rates of success. Let\u0026rsquo;s have a look at the various methods.\nBehavioral analysis Each system has processes running that consume resources like processor time and memory. As rootkits alter the execution path, its behavior might become visible.\nBy timing common operations (profiling), it can be possible to detect alterations in the kernel or system tools. You would then compare them with a known-good state. This known-good state can be measured from a similar system, or even from the system at a given time. When it is different at a later stage, then this might be a possible hint and may require additional research.\nAnother detection method is by creating files tailored to what rootkits may use. When they don\u0026rsquo;t show up in the output of a system tool or system call, it may indicate a rootkit at work.\nMemory analysis Although rootkits can be masters of illusion, they need to use disk storage, run in memory, or both. If a rootkit is using memory, then memory analysis can be used for detection. An open source memory forensics tool like Volatility can capture the contents of memory and analyze it.\nUsing signatures The antivirus industry uses fingerprints or signatures to detect common malware like viruses, worms, and backdoors. Some parts of the rootkit can be detected with the same approach. Many of the rootkits use hard-coded paths or names, making it easier to detect them.\nLog file analysis Suspicious events like daemons crashing could be a first indication of a system break-in. While it may not be directly related with a rootkit, it may be a hint. So monitoring the log files for unexpected events and crashes could be a useful step in the detection process.\nRootkit detection tools File integrity tools One method to detect alterations to a system is with the help of file integrity tools. These suites consist of several components:\nFile database Checksums Metadata Utilities The utilities are used to create and check checksums or fingerprints of files. They store the checksum in the database, together with metadata. Examples of metadata include the ownership, permissions, and timestamps of a file. Typically detection occurs at the moment when the current state is compared with an earlier moment in time. Previously it was Tripwire that was known to do this. Nowadays AIDE and Samhain are used for this purpose.\nRootkit scanners Specialized tools exist to detect traces of rootkits. These rootkit scanners search for common and uncommon files, compare the outputs of different utilities and try to trick a rootkit in revealing itself again. Rootkit Hunter and Chkrootkit are the most known tools.\nRootkit Hunter at work\nRootkits and false positives Most detection methods are not fool-proof. So-called false positives are common. This means that a finding is raised that actually is not related to the work by a rootkit. Tools try to reduce these false positives as much as possible. With the fine line between malicious software and just innocent system behavior, it is hard to be 100% secure.\nFrequently Asked Questions Is a rootkit harmful to the system? Yes, it is. A rootkit is malicious software and therefore unwanted malware.\nWhat is the best way to detect a rootkit? Use a malware scanner combined with file integrity monitoring and accounting of suspicious system calls. AIDE can be used for file integrity monitoring, complemented by the Linux Audit Framework to monitor changes and used system calls.\nHow can I remove a rootkit? As a rootkit is a master of illusion, it is almost impossible to remove it and being 100% sure of its complete removal. For that reason, the general advice is to do a fresh installation instead.\nWhich tools can I use to detect a rootkit? The detection of rootkits can be done by generic intrusion detection tools, malware scanners, or specific rootkit scanners for Linux .\nTools Some tools mentioned in this post:\nRootkit Hunter Chkrootkit OSSEC AIDE (Advanced Intrusion Detection Environment) Tripwire Did you find this article useful? Great! Make our community smarter and share this knowledge.\n","permalink":"https://linux-audit.com/intrusion-detection-linux-rootkits/","tags":["backdoor","intrusion detection","linux","malware","rootkit"],"title":"Detecting Linux rootkits"},{"categories":["Access Control List","Auditing","File Systems"],"contents":"Ever wondered what the plus (+) sign is when showing a directory listing? It is part of a POSIX standard to support access control lists (ACL) on files.\nNormal files on a file system will have only 10 characters displayed, with the last 9 used for file permissions. However, when file access control lists are used, an 11th character shows up. This plus sign indicates the usage of a file ACL.\ntotal 4 -rw-r-----+ 1 root root 5 May 29 14:36 test1 -rw-r----- 1 root root 0 May 28 11:52 test2 By using the command getfacl, the underlying permissions can be displayed. This command will display the normal file permissions, together with the more granular ones.\nThe use of getfacl/setfacl under Linux to apply file access control list.\nIn the screenshot, the user www-data has access to the file test1. This user is not listed in a group, nor being the owner (that is root). Still, this user has with the help of POSIX ACLs read access to the file.\nIf you never worked with ACLs, have a look at the man page of setfacl for some great examples. There will be a time when the normal file permissions are insufficient, yet you want to avoid using the \u0026ldquo;other\u0026rdquo; (everyone) group. POSIX ACLs to the rescue!\n","permalink":"https://linux-audit.com/plus-sign-ls-output/","tags":["ACL","file permissions","file system","getfacl","setfacl"],"title":"Plus sign in ls output"},{"categories":["nginx","Web"],"contents":"If you care about security, making your system \u0026ldquo;lean\u0026rdquo; is one very good start. Remove all clutter, like unused packages. It is part of system hardening and considered a good practice. This also applies to leaking of version numbers, which can only be harmful. Yes.. it is security through obscurity. But why would you reveal specific details about your environment to attackers? In this article we have a look at the very popular Nginx web server daemon.\nNginx version number Nginx shows the version number by default in error pages and in the headers of HTTP requests. For Nginx to hide this information, just a single statement is needed. Set the server_tokens statement to off in your global configuration file.\n# Don\u0026#39;t show the Nginx version number (in error pages / headers) server_tokens off; After making the changes, test your configuration.\nnginx -t\nNow restart your Nginx daemon. Next step is requesting a non-existing page. It should not display the Nginx version information anymore (just \u0026ldquo;Nginx\u0026rdquo;).\nservice nginx restart\nRemove \u0026ldquo;nginx\u0026rdquo; in output With the version gone, it still will show \u0026rsquo;nginx\u0026rsquo; in the output. If you want to remove this as well, additional steps might be needed.\nHex editor or manual compilation you may want to compile your nginx manually. Another option is to get creative and change the nginx binary with a hex editor. The downside is that these actions take a fair amount of time.\nRemove headers via reverse proxy If you are using a reverse proxy, you can leverage this to remove some of the headers as well. For example with Varnish you can decide to delete some of the headers by unsetting them.\nunset resp.http.X-Powered-By; unset resp.http.Server; Use the more_clear_headers There is another option, which is a function called more_clear_headers and part of the mod-headers package.\nOperating System Package Arch Linux nginx-mod-headers-more Red Hat nginx-module-headers-more Ubuntu libnginx-mod-http-headers-more-filter After installation of the module, use the function and tell it to clear the Server header.\nmore_clear_headers Server;\nNote: this line can be added in the http, server, location context. If you want to apply it for the whole server, add more_clear_headers in your /etc/nginx/nginx.conf file.\nLet\u0026rsquo;s test before the change is made and nginx is reloaded.\n# curl -I https://linux-audit.com HTTP/2 200 server: nginx date: Tue, 09 Apr 2024 07:34:53 GMT After the change, the Server header is gone.\n# curl -I https://linux-audit.com HTTP/2 200 date: Tue, 09 Apr 2024 07:37:20 GMT Automation Security auditing If you are responsible for many web servers, then we advise performing regular security audits. Vulnerability scanners can help here, like our open source (and free) auditing tool Lynis.\nConfiguration management Additionally, apply this nginx setting in a configuration management solution like Ansible, Cfengine, Chef, Puppet, or Salt. Every web server deployed will automatically have a more secure configuration.\n","permalink":"https://linux-audit.com/web/hiding-nginx-version-number/","tags":["hardening","information disclosure","nginx"],"title":"Hiding the nginx version number"},{"categories":["Auditing"],"contents":"Protecting computer networks consists of implementing preventative measures, but especially properly implementing detection methods. These digital tripwires can be used for intrusion detection, or proper handling security events on Unix systems.\nSecurity events First we have to define a few events which are or can be security related. To get easily started, we focus on 3 tips to implement security events on Unix systems.\n1. File changes Some files you don\u0026rsquo;t want to change that often, like your DNS resolvers stored in /etc/resolv.conf . An unexpected change to this file could indicate compromise. Similar of your password file, if the only user is your account and that of the root user.\nHow: Monitor these kind of security events can be done with a file integrity tool like AIDE, Samhain or Tripwire. Another measure is implementing an auditing framework, like the Linux audit framework.\n2. Process crashes Since usually software is a weak spot in security defenses, a crash of software often indicates an unexpected event. Sometimes caused by bad hardware (e.g. a bad memory module), but usually due to bad memory management in the software itself. Malicious people try to abuse these weak spots to load special crafted code. There monitoring crashing software can be very helpful to discover potential attacks or intrusions.\nHow: Check logging for processes which exited abruptly (segfault). Monitor also system uptime, as a system which is continuously crashing, is definitely not helping in availability requirements of the related business goal.\n3. Authentication Each system has a (business) goal. It is very common for a system to interact with users, in one way or another. Since most applications have authentication and authorization capabilities, properly monitoring these related events is important.\nHow: Set a threshold after which failures are logged. For example if a user enter his or her password once, ignore it. If it occurs three times in 1 minute, then create a security events. OpenSSH is one of the tools which sets such threshold by default. Related events should be picked up and monitored.\n","permalink":"https://linux-audit.com/audit-security-events-on-unix-systems/","tags":["auditing","linux","security"],"title":"Audit security events on Unix systems"},{"categories":["Access Control List","Linux"],"contents":"File ACLs can increase security due to the more granular permission structure. Still the use of ACLs is often not known to system administrators, resulting in directories and files having inappropriate file permissions.\nWhen to use (example) A directory could be configured with very tight permissions, including a proper owner and group. Normally the \u0026ldquo;Other\u0026rdquo; (everyone) group would have to be used to open up the file for people outside the owner group. This has a serious downside to open up a directory or file contents for all users.\nWith ACLs we can solve this issue. We still apply the tight permissions, however additionally we can give a single user file access.\nGetfacl To see the existing file permissions, the getfacl command can be used.\n# getfacl test1 # file: test1 # owner: root # group: root user::rw- group::r- other::- In this particular case the file has read-write permissions for the owner (root) and read access for the group (root).\nSetfacl When we want to allow the user \u0026ldquo;www-data\u0026rdquo; to access this particular file as well, we can adjust the ACL with the setfacl command.\nsetfacl -m u:www-data:r test1\nThis command adjust the file ACL and modified it to the user \u0026ldquo;www-data\u0026rdquo; having read access to the file, as can be seen in the screenshot below.\nThe use of getfacl/setfacl under Linux to apply file access control list.\nPlus sign in ls output If ACLs are applied to a file, the ls output will change. An additional plus sign will show up at the end of the line. This is to avoid overlooking the use of these additional permissions.\n# ls -l total 0 -rw-r--**+** 1 root root 0 May 28 11:52 test1 -rw-r-- 1 root root 0 May 28 11:52 test2 Conclusion File ACLs are powerful and provide a system administrator with more granular access possibilities. Where possible, use this feature to apply more strict file access, yet allowing the right people and process to access data.\n","permalink":"https://linux-audit.com/using-file-acls-linux-additional-security/","tags":["ACL","getfacl","setfacl"],"title":"Using File ACLs on Linux for Additional Security"},{"categories":["Auditing","Linux","Software"],"contents":"Proper software management is an important part in keeping your system secured. Acting on time is important, especially when network services have discovered security vulnerabilities.\nVulnerable packages Usually packages with known security vulnerabilities, get priority and updates are soon available. The risk in installing these packages is fairly low, as they don\u0026rsquo;t introduce new features. Instead, they fix the related security hole, which sometimes is nothing more than 1 single character!\nCheck your system Checking for vulnerable packages is a little bit tricky with the current version of the zypper command. However with the easy parse-able output of the \u0026ldquo;list packages\u0026rdquo;, we can extract all available package updates. From there we filter out only the packages marked as being security related.\nzypper lp | awk '{ if ($7==\u0026quot;security\u0026quot;) { if ($11==\u0026quot;update\u0026quot;) { print $13 } else { print $11 } } }' | sed 's/:$//' | grep -v \u0026quot;^$\u0026quot; | sort | uniq\nZypper then can be used to apply security updates (by package). Another option is to implement the output in a monitoring solution, especially for machines which require a high security level. The operations team then can quickly detect what systems need an audit for vulnerable packages.\nIf you want to automate checking, you could create a script and mail the output. Or use our security auditing tool Lynis to detect them. As vulnerable packages impose usually a high risk to the system, they will show up as warnings. Also the hardening index will decrease with each discovered package.\n","permalink":"https://linux-audit.com/vulnerabilities/audit-suse-with-zypper-vulnerable-packages/","tags":["linux","packages","software management","zypper"],"title":"Audit SuSE with zypper: vulnerable packages"},{"categories":["Auditing","Linux","Logging"],"contents":"By default the Linux audit framework logs all data in the /var/log/audit directory. Usually the related file is named audit.log and contains audit related information such as events.\n/var/log/audit/audit.log This file is the default log file for the Linux audit daemon. It has all related audit events and is configured using the configuration file of auditd (auditd.conf).\n# cat /etc/audit/auditd.conf log_file = /var/log/audit/audit.log Usually there is no reason to alter this location, unless a different storage location is preferred. For safeguarding of the data, it\u0026rsquo;s also wise to monitor this file and duplicate data to a locate storage location (e.g. with remote syslog).\nTools Although the log file is logged in plain ASCII format, it is better suited for parsing with specific tools in the audit framework. Two commands that come to mind are ausearch and aureport. Let\u0026rsquo;s have a look at both of them and how they work.\nausearch This utility helps with searching specific events, for example during the last day and with a specific type. Another possibility is filtering on a defined column, like a file name. In this case the file name should match the requested name.\naureport While searching for data has its purpose, reporting is a second useful goal. The aureport utility comes in handy and extracts all data from the file. It will then present the system administrator or auditor with the related information. This information varies from configuration changes (to the audit daemon) up to the amount of failed system calls (syscalls).\nSummary Report ====================== Range of time in logs: 12/07/2013 03:30:01.190 - 04/18/2014 15:00:01.378 Selected time for report: 12/07/2013 03:30:01 - 04/18/2014 15:00:01.378 Number of changes in configuration: 425 Number of changes to accounts, groups, or roles: 0 Number of logins: 0 Number of failed logins: 0 Number of authentications: 0 Number of failed authentications: 0 Number of users: 5 Number of terminals: 10 Number of host names: 0 Number of executables: 32 Number of files: 223 Number of AVC\u0026#39;s: 0 Number of MAC events: 0 Number of failed syscalls: 4190 Number of anomaly events: 3 Number of responses to anomaly events: 0 Number of crypto events: 0 Number of keys: 3 Number of process IDs: 31405 Number of events: 116468 The example above shows clearly how broad the events can be to be monitored. Proper configuration and safeguarding of the audit.log file will be necessary to protect this valuable information!\n","permalink":"https://linux-audit.com/linux-audit-log-files-in-var-log-audit/","tags":["audit","auditd","auditd.conf","rsyslog"],"title":"Linux audit: Log files in /var/log/audit"},{"categories":["Identity and Access Management","Linux"],"contents":"Like systems running Windows have an account named Administrator, Unix systems have their equal named \u0026ldquo;root\u0026rdquo;. This user with user id zero (0), have unlimited access to the system. Most applications implementing user access controls, apply a \u0026ldquo;backdoor\u0026rdquo; to allow this root user always access. This applies to access data, killing processes, starting kernel modules and more.\nTips to protect the root user Since the root user has unlimited access to the system, it make sense to protect this account carefully. Here are some tips..\n1. Make it difficult If you honestly care about your system security, then apply proper password rules to your root user account. Make sure it is long enough, have different character types and when possible even randomized.\n2. Different password per system You should avoid reusing passwords as much as possible. Especially with this administrative account, don\u0026rsquo;t repeat either.\n3. Change it regularly Use some tooling to change the password for each system on a regular basis. For example have one machine create random passwords, SSH into each machine and adjust the password. Then store the newly created passwords in a digital safe. Optionally with a hard-copy for you, or your manager.\n4. Use need-to-know principle Do really all system administrators need the root password for every machine? Or is it sufficient to apply a digital vault, in which a root password can only be accessed when really needed? If so, implement sudo and restrict passwords to a need-to-know basis.\n5. Monitor misuse People are lazy human beings. While you should be able to do your work activities, there is no excuse for compromising security just because it was \u0026ldquo;easier to work\u0026rdquo;. Monitor when and why the root user account is used.\n","permalink":"https://linux-audit.com/5-tips-to-protect-the-root-account/","tags":["authentication","linux","sudo"],"title":"5 Tips to protect the Root account"},{"categories":["Auditing","Hardening","Linux"],"contents":"System hardening is an important part in securing computer networks. Each system should get the appropriate security measures to provide a minimum level of trust. In this post we have a look at some of the options when securing a Red Hat based system. This information applies to Red Hat Linux (RHEL), Fedora, CentOS, Scientific Linux and others.\nRed Hat Red Hat itself has a hardening guide for RHEL 4 and is freely available.\nCIS The Center for Internet Security has guides, which are called \u0026ldquo;Benchmarks\u0026rdquo;. These benchmarks are available for the most popular operating systems, including Red Hat. While not always up-to-date with the latest release version, they provide valuable tips on securing your system. Some hardening snippets are included to automate the system hardening.\nNSA Also the NSA has a document created to hardening Red Hat. Unfortunately it\u0026rsquo;s outdated (RHEL 5), but might still be used to apply additional hardening measures on top of other guides. The PDF can be freely download.\nTools There aren\u0026rsquo;t many tools which help in auditing or hardening systems, which are also freely available and up-to-date. This is exactly the reason why we maintain Lynis and keep implementing new tests. Another big benefit of using a tool is automation. No hours of reading long pieces of text.\nSome alternatives are Tiger and Bastille Linux , which look both unmaintained at the moment. CIS has also a tool of their own CIS-CAT, but this is not open source and only available for companies having a membership with them.\nTips If you want to do an extensive check of your systems and implement proper hardening, then we advice to read the mentioned guides. Apply those principles which apply and appropriate for your environment. We argue that it\u0026rsquo;s better to use tooling and system automation though. It will save a lot of time, which can be invested in the actual system hardening.\n2015-01-30: Updated to later Fedora 18 guide (newer, but still outdated in version number)\n","permalink":"https://linux-audit.com/hardening-guides-tools-red-hat-linux-rhel/","tags":["auditing","hardening","tools"],"title":"Hardening Guides and Tools for Red Hat Linux (RHEL)"},{"categories":["Lynis"],"contents":"This week a vulnerability was reported in versions up to Lynis 1.5.4. With Lynis\nbeing a security audit tool and focused on hardening Linux and Unix based systems, we regret any (security) bug being discovered. Since it is open source software, we like to be open about the issue, to help you understanding it and take the right precautions.\nDescription:\nThe temporary files created in the tests_webservers section are too predictable.\nThis may resulting in a possible race condition, where a local user creates the\ntemporary file and symlinks it to an existing system file. Lynis then uses this\nfile to store temporary data. As a result data is overwritten in the (linked) file.\nAdvice:\nYou are advised to upgrade Lynis to at least version 1.5.5, which has adjustments\nto counter the vulnerability.\nWorkarounds:\nRemove the temporary file creation in tests_webservers and disable the related tests using the temporary files.\nRisks:\nThe chance for exploitation is considered low. The following conditions have to apply:\nLynis has to be executed at that moment (usually once a day, or less). Access to the system is needed to the temporary file (to create file and guess the right name) Perfect timing of creating the symbolic link, as the window of opportunity is very small. Related information:\nUnfortunately this bug was not reported according common rules of responsible disclosure.\nThis resulted in two different CVE entries where created.\nCVE-2014-3982: AIX\nCVE-2014-3986: Linux and others (except AIX)\nWe are sorry for any inconvenience and will use this blog post as the main article to provide any further updates.\n","permalink":"https://linux-audit.com/lynis-security-notice-154-and-older/","tags":["lynis","one-time"],"title":"Lynis Security Notice: 1.5.4 and older"},{"categories":["Auditing","Forensics","Linux","System Administration"],"contents":"From the initial start of the Linux operating system, the first processes are already born. In this article we have a look on dealing with processes. In particular we look at how to do process auditing. Whenever you are an auditor, system administrator or just a Linux enthusiast, you can\u0026rsquo;t ignore processes and should know how to deal with them.\nProcess listing For most people working on Linux systems, it might be obvious to display running processes with ps. For Linux it\u0026rsquo;s common to use ps -ef, which shows effectively a list of all processes with a full listing. Those who are used to work on BSD machines will prefer using ps aux. On Linux with the POSIX tools, both will work, however with a slightly different output.\nParent process Every process, except init has a parent process. This is the process which started another one. Usually when a program consists of only one process (no children), it will be spawned with init as its parent process. In such case the PPID column of ps will show the ID value 1.\n# ps -ef UID PID PPID C STIME TTY TIME CMD root 16343 1 0 Apr14 ? 00:00:00 nginx: master process /usr/sbin/nginx www-data 16344 16343 0 Apr14 ? 00:00:13 nginx: worker process www-data 16345 16343 0 Apr14 ? 00:00:12 nginx: worker process www-data 16346 16343 0 Apr14 ? 00:00:16 nginx: worker process www-data 16347 16343 0 Apr14 ? 00:00:14 nginx: worker process From this example we can clearly see that there is one master process (PID 16343, parent: init) and having 4 children (worker processes).\n/proc file system The virtual /proc file system provides us with additional information about the kernel and running processes. While most of the information can be extracted via ps or other tools, the information in /proc is easily accessible. Let\u0026rsquo;s dive into a few common files and their information treasure.\n/proc/pid/cmdline Most processes will be started with some parameters. If so, these parameters will be listed in the cmdline file. If there are no parameters, the binary itself will be displayed.\n/proc/pid/exe The exe file is a symlink to the actual binary on disk. In case some process is running, this might help finding the related binary.\n/proc/pid/fd/ Most processes need to open a file or a socket. This happens with a system call like fopen, to open a file descriptor (fd). The fd directory within the /proc file system shows all file descriptors. When displaying a file listing, the related files (or sockets) will be displayed.\n# ls -l /proc/16343/fd/ total 0 lrwx-- 1 root root 64 Apr 18 18:07 0 -\u0026gt; /dev/null lrwx-- 1 root root 64 Apr 18 18:07 1 -\u0026gt; /dev/null lrwx-- 1 root root 64 Apr 18 18:07 10 -\u0026gt; socket:[1310432] lrwx-- 1 root root 64 Apr 18 18:07 11 -\u0026gt; socket:[1310433] lrwx-- 1 root root 64 Apr 18 18:07 12 -\u0026gt; socket:[1310834] lrwx-- 1 root root 64 Apr 18 18:07 13 -\u0026gt; socket:[1310835] lrwx-- 1 root root 64 Apr 18 18:07 14 -\u0026gt; socket:[1310836] lrwx-- 1 root root 64 Apr 18 18:07 15 -\u0026gt; socket:[1310837] lrwx-- 1 root root 64 Apr 18 18:07 16 -\u0026gt; socket:[1310838] lrwx-- 1 root root 64 Apr 18 18:07 17 -\u0026gt; socket:[1310839] lrwx-- 1 root root 64 Apr 18 18:07 18 -\u0026gt; socket:[1310840] l-wx-- 1 root root 64 Apr 18 18:07 19 -\u0026gt; /var/log/nginx/cisofy.local.access.log l-wx-- 1 root root 64 Apr 18 18:07 2 -\u0026gt; /var/log/nginx/error.log lr-x-- 1 root root 64 Apr 18 18:07 3 -\u0026gt; /proc/16342/auxv lrwx-- 1 root root 64 Apr 18 18:07 4 -\u0026gt; socket:[1310833] l-wx-- 1 root root 64 Apr 18 18:07 6 -\u0026gt; /var/log/nginx/access.log l-wx-- 1 root root 64 Apr 18 18:07 7 -\u0026gt; /var/log/nginx/error.log l-wx-- 1 root root 64 Apr 18 18:07 8 -\u0026gt; /var/log/nginx/localhost.access.log lrwx-- 1 root root 64 Apr 18 18:07 9 -\u0026gt; socket:[1310431] /proc/pid/fdinfo/ Within the fdinfo subdirectory, we can additionally find more information about the file descriptor itself.\n# cat /proc/16343/fdinfo/3 pos: 304 flags: 0100000 Pos defines the position, where flags describe the related \u0026ldquo;parameters\u0026rdquo; used when opening the file (read-only, append, write, etc).\n/proc/pid/syscall The syscall file is available with newer kernels. It displays the current status by sharing the system call it last performed. Knowing the most important system calls is valuable for auditing purposes. For example they are also used when monitoring file access or other system events, together with the Linux audit framework.\nDisplaying the output of the file might look like:\n# cat /proc/16343/syscall 130 0x7fff714df300 0x8 0x0 0x0 0xcccccccd 0x7ff6148d7400 0x7fff714df2a8 0x7ff6147cb77a Well, this doesn\u0026rsquo;t give much information at a first glance, except that the first identifier is the syscall ID. In this case we can easily lookup number 130 by using the ausyscall tool (if installed).\n# ausyscall 130 \u0026gt; rt_sigsuspend Alternative option: For x86_64 based systems, look for the file unistd_64.h and grep for the related ID. Make sure to determine the proper machine type with uname -m.\n/proc/id/stack Another option to determine the latest system call is by checking the stack. This file (/proc/pid/stack) will display something like:\n# cat /proc/16343stack** [\u0026lt;ffffffff8107f269\u0026gt;] sys_rt_sigsuspend+0x89/0xc0 [\u0026lt;ffffffff81669b82\u0026gt;] system_call_fastpath+0x16/0x1b [\u0026lt;ffffffffffffffff\u0026gt;] 0xffffffffffffffff The top row displays the current system call and should be similar to what is in the syscall file.\nAs can be seen, there is a lot of information available about processes.\n","permalink":"https://linux-audit.com/auditing-linux-processes/","tags":["auditing","forensics","linux","processes"],"title":"Auditing Linux processes: The Deep Dive!"},{"categories":["Software"],"contents":"How it works Each time the SSH client connects with a server, it will store a related signature (a key) of the server. This information is stored in a file names named known_hosts. The known_hosts file itself is available in the .ssh subdirectory of the related user (on the client). In the case the signature of the server changes, SSH will protect the user by notifying about this chance.\nRisk involved This configuration option is very useful, but also introduces a new risk. Previously it was common to store the hostname related with the key. The result is a \u0026ldquo;picture\u0026rdquo; of the network, revealing which systems are connected. This made it easy for worms and other malicious scripts to use this information and spread to other systems, once they had a single system compromised.\nImprove security To reduce the risk of storing a clear picture of the network, the solution introduced was hashing the hostname. To enable this functionality, the HashKnownHosts option can be set to yes. This option can be found in the system-wide SSH client configuration file, which is usually /etc/ssh/ssh_config.\nThe final result of hashing entries will look something like this:\n|1|XV5CFMH8LLIQPq7PxdBhGX7I9PA=|VKNLdODsQlJ/j4cvTZncqs9vgh0= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKuhGhv+2AUY2IapdqToiZgCDOnBNT3dbnFL79FQ0JofFmxE9b/jqlwN+a7ZPKsmf+UdJ/RzzZLH8Hs0UgroC0= The hostname (hashed with ecdsa-sha2-nistp256) is unreadable for the human eye or malicious scripts. For each new connection to the related host, the hashing algorithm will result in the same string. This way the client knows it already has a stored key and compare it during the handshaking process with the server.\n","permalink":"https://linux-audit.com/ssh/audit-ssh-configurations-hashknownhosts-option/","tags":["ssh","system hardening"],"title":"Audit SSH configurations: HashKnownHosts option"},{"categories":["Hardening"],"contents":"Most systems have confidential data that needs to be protected. To safeguard this data, we need to secure our Linux system. But how to properly harden a Linux system? In this article, we will cover this step by step. We start by with physical security measures to prevent unauthorized people from access the system in the first place. Next is doing the installation the right way, so we have a solid foundation. Finally, we will apply a set of common security measures. After we are finished, your server or desktop system should be better protected. Are you ready? Let\u0026rsquo;s proceed with the first steps!\nLinux is already secure by default, right? One of the myths about Linux is that it is secure, as it is not susceptible to viruses or other forms of malware. This is partially true, as Linux uses the foundations of the original UNIX operating system. Processes are separated and a normal user is restricted in what he or she can do on the system. Still, Linux is not perfectly secure by default. One of the reasons is the Linux distributions that package the GNU/Linux kernel and the related software. They have to choose between usability, performance, and security.\nWith the difficult choices that Linux distributions have to make, you can be sure of compromises. These compromises typically result in a lowered level of security. What about malware for Linux? That is a definitely a myth. The Linux platform also has its fair share of backdoors, rootkits, works, and even ransomware. That is one of the reasons why it is important to do system hardening, security auditing, and checking for compliance with technical guidelines.\nThere are many aspects to Linux security, including Linux system hardening, auditing, and compliance.\nWhat is system hardening? To improve the security level of a system, we take different types of measures. This could be the removal of an existing system service or uninstall some software components.\nSystem hardening is the process of doing the \u0026lsquo;right\u0026rsquo; things. The goal is to enhance the security level of the system. There are many aspects to securing a system properly. Yet, the basics are similar for most operating systems. So the system hardening process for Linux desktop and servers is that that special.\nCore principles of system hardening If we would put a microscope on system hardening, we could split the process into a few core principles. These include the principle of least privilege, segmentation, and reduction.\nPrincipe of least privilege The principle of least privileges means that you give users and processes the bare minimum of permission to do their job. It is similar to granting a visitor access to a building. You could give full access to the building, including all sensitive areas. The other option is to only allow your guest to access a single floor where they need to be. The choice is easy, right?\nExamples:\nWhen read-only access is enough, don\u0026rsquo;t give write permissions Don\u0026rsquo;t allow executable code in memory areas that are flagged as data segments Don\u0026rsquo;t run applications as the root user, instead use a non-privileged user account Segmentation The next principle is that you split bigger areas into smaller ones. If we look at that building again, we have split it into multiple floors. Each floor can be further divided into different zones. Maybe you visitor is only allowed on floor 4, in the blue zone. If we translate this to Linux security, this principle would apply to memory usage. Each process can only access their own memory segments.\nReduction This principle aims to remove something that is not strictly needed for the system to work. It looks like the principle of least privilege, yet focuses on preventing something in the first place. A process that does not have to run, should be stopped. Similar for unneeded user accounts or sensitive data that is no longer being used.\nSystem hardening steps Overview of hardening steps\nInstall security updates and patches Use strong passwords Bind processes to localhost Implement a firewall Keep things clean Security configurations Limit access Monitor your systems Create backups (and test!) Perform system auditing 1. Install security updates and patches Most weaknesses in systems are caused by flaws in software. These flaws we call vulnerabilities. Proper care for software patch management help with reducing a lot of the related risks. The activity of installing updates often has a low risk, especially when starting with the security patches first. Most Linux distributions have the option to limit what packages you want to upgrade (all, security only, per package). Make sure that your security updates are installed as soon as they come available. It goes without saying, before you implementing something, test it first on a (virtual) test system.\nDepending on your Linux distribution there might be a way to implement security patches automatically, like unattended upgrades on Debian and Ubuntu. This makes software patch management a lot easier!\n2. Use strong passwords The main gateway to a system is by logging in as a valid user with the related password of that account. Strong passwords make it more difficult for tools to guess the password and let malicious people walk in via the front door. A strong password consists of a variety of characters (alphanumeric, numbers, special like percent, space, or even Unicode characters).\n3. Bind processes to localhost Not all services have to be available via the network. For example, when running a local instance of MySQL on your web server, let it only listen on a local socket or bind to localhost (127.0.0.1). Then configure your application to connect via this local address, which is typically already the default.\n4. Implement a firewall Only allowed traffic should in an ideal situation reach your system. To achieve this, implement a firewall solution like iptables, or the newer nftables.\nWhen creating a policy for your firewall, consider using a \u0026ldquo;deny all, allow some\u0026rdquo; policy. So you deny all traffic by default, then define what kind of traffic you want to allow. This is especially useful for incoming traffic, to prevent sharing services you didn\u0026rsquo;t intend to share.\nUseful reads:\nDifferences between iptables and nftables 5. Keep things clean Everything installed on a system which doesn\u0026rsquo;t belong there can only negatively impact your machine. It will also increase your backups (and restore times). Or they might contain vulnerabilities. A clean system is often a more healthy and secure system. Therefore minimization is a great method in the process of Linux hardening.\nActionable tasks include:\nDelete unused package Clean up old home directories and remove the users 6. Secure configurations Most applications have one or more security measures available to protect against some forms of threats to the software or system. Look at the man page for any options and test these options carefully.\n7. Limit access Only allow access to the machine for authorized users. Does someone really need access or are alternative methods possible to give the user what he or she wants?\n8. Monitor your systems Most intrusions are undetected, due to lack of monitoring. Implement normal system monitoring and implement monitoring on security events. For example, the use of the Linux audit framework increased detection rates of suspected events.\n9. Create backups (and test!) Regularly make a backup of system configuration and its data. This can prevent data loss and help restoring systems after a disaster. Even more important, perform a restore and test those backups. Having a backup is nice, but it is the restore that really counts!\nBackups can be done with existing system tools like tar and scp. Another option to spare bandwidth is synchronizing data with tools like rsync.\nOther options:\nAmanda Bacula Duplicity Syncthing 10. Perform system auditing You can\u0026rsquo;t properly protect a system if you don\u0026rsquo;t measure it.\nUse a security tool like Lynis to perform a regular audit of your system. Any findings are showed on the screen and also stored in a data file for further analysis. With an extensive log file, it allows to use all available data and plan next actions for further system hardening.\nLynis runs on almost all Linux systems or Unix flavors. It only requires a normal shell. Root permissions are preferred, yet not needed. The security tool is free to use and open source software (FOSS).\nSecurity tools for Linux There are so many security tools that work on Linux, that we created a dedicated website. Have a look at the Linux security tools top 100 to learn what type of tools there are.\nAdditional hardening resources Ready for more system hardening? Read then the extended version of the Linux security guide.\n","permalink":"https://linux-audit.com/linux-server-hardening-most-important-steps-to-secure-systems/","tags":["hardening","linux","linux security","ransomware","server hardening","system hardening"],"title":"Linux server hardening and best practices"},{"categories":["Lynis"],"contents":"Keeping software like Lynis up-to-date is nowadays very important. More and more vendors implement software development methodologies like agile and scrum, to decrease the time between new software versions. This way software enhancements are easier to implement and possible bugs earlier fixed. It\u0026rsquo;s up to the user of the software to stay up-to-date and therefore we provide some tips on how to update Lynis easily.\nNotifications Staying up-to-date begins with receiving an update when a new release is available. For Lynis there are several possibilities, so everyone can use it\u0026rsquo;s preferred method.\nSocial media Company founder Michael Boelen can be followed via Mastodon at @mboelen .\nLynis Another way to determine if there is an update, is using Lynis itself and use the -check-update parameter. It will display \u0026ldquo;Up-to-date\u0026rdquo; if Lynis is running the latest version.\n# ./lynis -check-update == Lynis == Version : 1.4.8 [ Up-to-date ] Release date : 27 March 2014 Update location : http://cisofy.com Copyright 2007-2014 - Michael Boelen, http://cisofy.com Download New releases of Lynis and plugins can be downloaded via the CISOfy website . Before running new software versions, you are advised to check the related SHA1 checksum first. This hash is published on the download page, together with the link to the Lynis version itself.\nPackages When using a Lynis package, it usually takes time before the latest version is available. Depending on the platform you are using, a request for an update can be requested (if it\u0026rsquo;s really outdated). For example when using FreeBSD, a PR could be created to update the ports tree entry for Lynis.\nWhen using custom build packages, updating might be as simple as altering the version number and rebuilding the package.\n","permalink":"https://linux-audit.com/lynis/how-to-keep-lynis-up-to-date/","tags":["lynis"],"title":"How to keep Lynis up-to-date?"},{"categories":["Auditing","Lynis"],"contents":"Want to know the vulnerabilities of a Unix/Linux system is in just 3 minutes? How? Perform a scan with Lynis, the open source Unix security audit tool!\nLynis Lynis is open source software (GPLv3), released in 2007 and a popular choice by many security professionals and system administrators. Hundreds of downloads in the first week of each release and with a lot of community feedback, Lynis is the right tool for the job.\nQuick start: installation Install Lynis via apt-get install lynis or yum install lynis. Or if you want to use the very latest version, go to the CISOfy website, click on products and select Lynis. Download the file (e.g. with wget) and extract the tar ball.\nQuick start: running Time to run Lynis! As we want to stay under the 3 minutes, let\u0026rsquo;s run Lynis with the command audit system\nlynis audit system\nThis will use the default scan profile and perform all tests without any pauses. The screen will display the output results directly. After all tests are done, a quick overview will be given with the findings (warnings or suggestions).\nAdditionally a hardening index will be displayed, giving a first impression on how well the system is hardened. If the bar is colored red, then the system really requires some attention. For yellow and green colored bars it\u0026rsquo;s advised to follow-up on the displayed findings and determine the related risks.\nFollow-up After the first scan is done, the next step would be to actually review the results in-depth. Of course this process will take more time. One proper way of determining what has been scanned and discovered, is by checking the log file. By default the file /var/log/lynis.log is being used.\nless /var/log/lynis.log\nNow scroll through the file and check the results of each particular test.\nThree minutes See? Running a Unix security audit doesn\u0026rsquo;t have to take that long. Good luck with hardening!\nDon\u0026rsquo;t know where to start with hardening, or how? See our Lynis Enterprise Suite offering, it will help you by showing an implementation plan with priorities. For enterprise users we also have hardening snippets available, for easier implementation.\n","permalink":"https://linux-audit.com/unix-security-audit-perform-an-audit-in-3-minutes/","tags":["audit","lynis"],"title":"Unix security audit: Perform an audit in 3 minutes"},{"categories":["Auditing","Linux"],"contents":"Now open source software and platforms are very common, the need for knowledge in this area is increasing. Becoming a technical auditor with specialized knowledge about Linux, might be a clever move.\nTechnical When specializing in Linux, the auditing area is already more technically oriented, instead of the processes. A true Linux auditor knows more than the basics of Linux. In-depth knowledge is required, like what file systems are common, how permissions are arranged, popular applications are common (at the presentation layer, middleware, backend).\nApplications Since most applications are used over and over, focusing on those is very helpful. Think about Apache, MySQL, PHP (LAMP stack), but also emerging alternatives. For example Nginx is a nice example of software taking its share in the area of web servers.\nUsers and Permissions For each system where users can log in and auditor should be able to know who they are and what they can do. Not personally of course, but regarding identity and access management (IAM)\nNetwork processes To determine additional risks, focusing on network communications is useful. Starting with all daemons listening on a network interface and zooming in on server configurations. Additionally having knowledge and experience with iptables (or alternatives) is of value for one being a real Linux auditor.\nCertifications The CISA (Certified Information Systems Auditor) certification from ISACA is the one you definitely should have. It provides the basics and more of the auditing profession. Additionally technical certifications will be useful, like the ones from LPI (LPIC-1, LPIC-2 and LPIC-3). Also Comptia (Linux+) is an alternative, but more generic. For specialization in Red Hat, one might even become RHCE to truly understand how Linux systems, with Red Hat in particular, are working.\nTools Knowing your tools is usually the key of making your life easier. Why do everything manually when specialized tools can do the trick? Use port scanners like nmap to scan the network, IDS set-ups like Snort to monitor for suspicious traffic. What about the auditing tool for a Linux auditor? Of course, our tool Lynis to help you performing an in-depth scan of Linux security.\nMany pentesting distributions will be of help for seasoned Linux professionals, by combining all powerful tools into one system. No need for manual installation, as many tools are already installed and grouped per category.\nHappy auditing!\n","permalink":"https://linux-audit.com/linux-auditor-what-to-know/","tags":["auditor","linux","pentest","penetration testing"],"title":"Become a Linux Auditor: What to know?"},{"categories":["Auditing","Linux"],"contents":"In this article we answer the big question on Linux systems \u0026ldquo;what to audit?\u0026rdquo;. Where do you start and what is useful to audit? We apply our three C\u0026rsquo;s in this article to determine what we should look for when auditing a Linux system.\nCurrent state What is the current state of the system and how does it compare to previous time?\nIdeal situation: compare current state of the system with a predefined baseline or previous scan\nRunning an audit on a Linux system might reveal flaws in its configuration. While this is useful information to know, it\u0026rsquo;s even more valuable to know since when this flaw is present (past) and how to act upon it (future). Therefore companies should have a security policy defining appropriate measures and actions. In addition it should be accompanied by the right (technical) procedures, to define how systems should be configured.\nBaselines\nTo audit a Linux system one might start by auditing configuration files first. Most of these can be found under the /etc directory and its sub-directories. Finding changes is easy by comparing the full directory and determine what lines are changed in each file. Saving a baseline might give especially useful insights when changes are found, but can not be explained (e.g. loading a new kernel module).\nAnother important item is to save earlier performed audits, to check the state of systems in the past and what measures are taken on a particular system. Especially when dealing with exceptions to the security policy, the documentation should be carefully updated. It should also include the appropriate permission and bookkeeping why this exception is tolerated and for how long.\nAutomation\nFor Linux and Unix based systems we created our open source (GPLv3) tool Lynis. It can be configured to perform tests from some categories, or all of them. To allow exceptions, particular tests can be excluded. The tools allows to create a report file, making it easier to determine the state of the system and comparing it with other scans.\nChanges to the system What changes occurred to the system?\nIdeal situation: no changes, or only authorized and well-documented changes\nTracking changes over time is a challenging exercise on most systems. That\u0026rsquo;s why continuous audits are needed to discover (unauthorized) changes quickly. Usually it\u0026rsquo;s easier to act upon a recent change, compared to change in a configuration file from two years ago.\nChanges should be monitored and also properly documented. For Linux systems this could be done in the configuration file itself, or in the central repository of a configuration management solution. Another common method for companies using ITIL is in the change management tooling.\nOn Linux changes to critical configuration files could be easily detected by implementing the Linux audit framework. We have already written some articles about this subject in the past and are worth reading as well. The framework is powerful and can detect authorized changes, but also security intrusions.\nControl access What access do users have and to what?\nIdeal situation: clear matrix on which users can access the system and what access they have\nSimilar to having baselines on a system configuration, it is very useful to have an access matrix. This list of people and their type of access to the system, can be used for easily auditing permissions at a later stage. Unfortunately it\u0026rsquo;s not common yet that this information is properly documented (and updated!).\nRegular audit of the passwd file and changes to it, will provide insights to whom can access the system. To discover what data can be actually accesses, a file system scan should be performed.\n","permalink":"https://linux-audit.com/auditing-linux-what-to-audit/","tags":["audit","auditing","linux"],"title":"Auditing Linux: what to audit?"},{"categories":["Auditing","Logging"],"contents":"This guide is to help our users of the Lynis Enterprise Suite to configure a central node to receive Linux audit events. It provides some pointers on how to do a quick set-up, to store and forward events. This information is very valuable for forensic investigations and intrusion detection.\nConfigure the server First start by configuring the server. Since this is a central log host, it should have enough disk capacity and enough bandwidth to sustain peaks.\nFor these examples we use the rsyslog server. It\u0026rsquo;s commonly available on Linux distributions and a very powerful syslog daemon, with flexibility in mind.\n/etc/rsyslog.conf\n# Receive syslog messages via TCP $ModLoad imtcp $InputTCPServerRun 514 $AllowedSender TCP, 127.0.0.1, 192.168.1.0/24, 1.2.3.4 Restart the rsyslog daemon and see if it now listens to port 514\nss -plant | grep 514\nSend a test message from a client system\nlogger -p local6.info --tcp -P 514 --server 192.168.1.10 test\nIf that works correctly, let\u0026rsquo;s further tune the configuration to allow for a custom file format.\n$template HostAudit, \u0026#34;/var/log/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%_audit.log\u0026#34; $template auditFormat, \u0026#34;%msg%\\n\u0026#34; local6.* ?HostAudit;auditFormat The ModLoad enables the reception of syslog messages. Depending on your set-up, you can also alter the default listening port (514) and limit allowed senders. The format (HostAudit) creates a directory structure for the audit files, by date and puts data in a file specifically by hostname. To avoid cluttering up files, hostnames should therefore be unique.\nConfigure the clients On the client we have to adjust the rsyslog configuration to perform two tasks:\nMonitor the audit file Log all events to the central node /etc/rsyslog.conf\n# Add under the modules section $ModLoad imfile # Add at the end of the file # Add at bottom of configuration file $InputFileName /var/log/audit/audit.log $InputFileTag tag\\_audit\\_log: $InputFileStateFile audit_log $InputFileSeverity info $InputFileFacility local6 $InputRunFileMonitor local6.* @@192.168.1.10:514 Permissions Make sure to adjust the group membership of /var/log/audit and underlying files to syslog. This way the rsyslog daemon can actually read the files. After adjusting the permissions, reload rsyslog and check for any warnings in /var/log/syslog.\nTesting On the client, use cat of a file which is being watched (in our case /home/cisofy/test).\nThis should create an event on the client system itself, in the /var/log/audit/audit.log file. We can test this with the command ausearch -k test.\nNext check is to see if the data has been logged on both the client system and remote syslog server.\nroot@server:/var/log/rsyslog/2014/03/19# ausearch -if cisofy1_audit.log -k test time-\u0026gt;Wed Mar 19 19:08:46 2014 type=PATH msg=audit(1395252526.691:8194): item=0 name=\u0026#34;test\u0026#34; inode=411440 dev=fd:03 mode=0100664 ouid=1002 ogid=1002 rdev=00:00 type=CWD msg=audit(1395252526.691:8194): cwd=\u0026#34;/home/cisofy\u0026#34; type=SYSCALL msg=audit(1395252526.691:8194): arch=c000003e syscall=90 success=yes exit=0 a0=1ae23a0 a1=81b4 a2=0 a3=0 items=1 ppid=4725 pid=31097 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=2690 comm=\u0026#34;vi\u0026#34; exe=\u0026#34;/usr/bin/vim.tiny\u0026#34; key=\u0026#34;test\u0026#34; ","permalink":"https://linux-audit.com/central-audit-logging-configuration-collecting-linux-audit-events/","tags":["audit","logging","rsyslog","syslog"],"title":"Configuration and collecting of Linux audit events"},{"categories":["Auditing","Linux","Lynis"],"contents":"This guide helps people new to the Linux platform to get a grasp on how the system works. Whenever you are an IT auditor, or simply want to know more about the basics, this guide helps you in determining where to start an audit.\nProcesses Each operating system consists of smaller running processes. In case of Linux this is true as well and can be displayed with the ps tool. Without parameters it will already show some processes, but the list is not complete. To see a full list of running processes, use ps -ef or ps aux.\nUsers What would a system be without any users using it? To get a first hint on what users can access the system, check the /etc/passwd file. Additionally the /etc/shadow file will have similar data, including hashed passwords.\nName services o see what sources are used for name services like DNS, check the contents of the /etc/nsswitch.conf file.\n# **cat /etc/nsswitch.conf** # /etc/nsswitch.conf # # Example configuration of GNU Name Service Switch functionality. # If you have the \u0026#39;glibc-doc-reference\u0026#39; and \u0026#39;info\u0026#39; packages installed, try: # \u0026#39;info libc \u0026#34;Name Service Switch\u0026#34;\u0026#39; for information about this file. passwd: compat group: compat shadow: compat hosts: files dns networks: files protocols: db files services: db files ethers: db files rpc: db files netgroup: nis Data storage Most systems store or process data locally. To determine where data might be present, there are a few utilities handy to help:\n/etc/fstab file\nThe fstab file contains common mount points. While it doesn\u0026rsquo;t display all mount points, it\u0026rsquo;s a good start to see what file systems are available.\n# cat /etc/fstab # /etc/fstab: static file system information. # # Use \u0026#39;blkid -o value -s UUID\u0026#39; to print the universally unique identifier # for a device; this may be used with UUID= as a more robust way to name # devices that works even if disks are added and removed. See fstab(5). # # \u0026lt;file system\u0026gt; \u0026lt;mount point\u0026gt; \u0026lt;type\u0026gt; \u0026lt;options\u0026gt; \u0026lt;dump\u0026gt; \u0026lt;pass\u0026gt; proc /proc proc defaults 0 0 # / was on /dev/sda2 during installation UUID=7c61eaaa-9d7f-4b17-82a3-99699f331073 / ext4 errors=remount-ro 0 1 # swap was on /dev/sda3 during installation UUID=e3891483-faed-499d-80e2-0dd7331118cd none swap sw 0 0 /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0 /dev/sdb1 /mnt/removable-disk ext4 noauto,noexec 0 0 mount\nMount displays the active file systems and so called mount points. Commonly a /data mount is available. If not and the root (/) is very big, it might be directory on this system. Another common location is /usr/local, or an external mount point (e.g. NFS server).\n# mount /dev/sda2 on / type ext4 (rw,errors=remount-ro) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) udev on /dev type devtmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev) rpc\\_pipefs on /run/rpc\\_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) lsof\nTo see all open files, revealing more hints on where data is stored, use the lsof tool (list open files).\nCOMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\ninit 1 root cwd DIR 8,2 4096 2 /\ninit 1 root rtd DIR 8,2 4096 2 /\ninit 1 root txt REG 8,2 167192 136565 /sbin/init\ninit 1 root mem REG 8,2 52120 1975156 /lib/x86_64-linux-gnu/libnss_files-2.15.so\ninit 1 root mem REG 8,2 47680 1975160 /lib/x86_64-linux-gnu/libnss_nis-2.15.so\ninit 1 root mem REG 8,2 97248 1975172 /lib/x86_64-linux-gnu/libnsl-2.15.so\ninit 1 root mem REG 8,2 35680 1975154 /lib/x86_64-linux-gnu/libnss_compat-2.15.so\ninit 1 root mem REG 8,2 1815224 1975152 /lib/x86_64-linux-gnu/libc-2.15.so\ninit 1 root mem REG 8,2 31752 1975159 /lib/x86_64-linux-gnu/librt-2.15.so\ninit 1 root mem REG 8,2 135366 1975161 /lib/x86_64-linux-gnu/libpthread-2.15.so\ninit 1 root mem REG 8,2 276392 1966987 /lib/x86_64-linux-gnu/libdbus-1.so.3.5.8\ninit 1 root mem REG 8,2 38888 1967542 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0\ninit 1 root mem REG 8,2 96240 1967544 /lib/x86_64-linux-gnu/libnih.so.1.0.0\ninit 1 root mem REG 8,2 149280 1975164 /lib/x86_64-linux-gnu/ld-2.15.so\ninit 1 root 0u CHR 1,3 0t0 5793 /dev/null\ninit 1 root 1u CHR 1,3 0t0 5793 /dev/null\ninit 1 root 2u CHR 1,3 0t0 5793 /dev/null\nInstalled software No system can run without software. Linux based systems do often have pre-installed packages to form a minimal base. Common examples are tools like grep, cut and awk. Additionally, network based services might be installed during the installation.\nDebian/Ubuntu: dpkg -l\nRedHat/CentOS: rpm -qa\nAutomation While manually checking traces on a system is fine, automation is even better. It saves time, effort and improves the quality of an audit. Where needed, manual checks can still be an extension of an automated audit. Combining both will improve the audit even more.\nWithin this range of articles we already shared our tool Lynis. In case you didn\u0026rsquo;t use it yet, this might be the time to become an auditor in only a few minutes. Lynis is free and open source, used by many professionals and contains over 250 individual tests. Consider only the time if you would need to check everything manually!\nWe hope this article gave you some new insights. Want to know more about a particular subject? Let us known via the about section!\n","permalink":"https://linux-audit.com/become-a-linux-auditor-tips-to-start-with-auditing-the-linux-platform/","tags":["auditor","linux"],"title":"Become a Linux auditor: tips to start with auditing the Linux platform"},{"categories":["Lynis"],"contents":"Within this \u0026ldquo;how to\u0026rdquo; we explain when and how to use Lynis plugins.\nWhat are plugins? Plugins are small extensions to an existing program. Also Lynis supports the use of external plugins to extend functionality. Lynis plugins are written in shell script and might use system binaries or external binaries to perform additional checks.\nThe big difference between custom tests and plugins in Lynis, are the goal of the tests. If some logic function checks a value and can inform the user to take an action, it\u0026rsquo;s better to use a normal test. For tests which primarily gather information, to be processed later, then a plugin is better suited.\nUsually test data from a plugin will be reported in the log file ( /var/log/lynis.log ) and the report. Of course it depends on the plugin itself on how much data will be stored.\nCreating a plugin Create a file by using the plugin template file. Give it a filename plugin_PLUGINNAME_phase1 and limit the permissions (e.g. chmod 600). If you create a custom plugin, we suggest to add custom to it (e.g. plugin_mycompany-custom_phase1), to prevent filename clashes.\nNow alter the file and set the author, plugin name (short name) and version number. The category is especially useful for bundled plugins. If there is only one plugin, give it a name you prefer (e.g. custom).\nNext thing is to activate the plugin in the scan profile. Add a line \u0026ldquo;plugin=pluginname\u0026rdquo;, so Lynis knows it should actually use the plugin.\nWhen creating tests, don\u0026rsquo;t use existing test IDs or categories. Use the CUST category with a followup number (CUST-001, CUST-002 etc), so it\u0026rsquo;s clear that the tests are custom build to your needs.\nCustom plugins Depending on your needs, you might want to create a custom plugin. While we support that, please note that often it\u0026rsquo;s not needed. Many tests can be integrated in the main Lynis code. For users of the Enterprise Suite we can even help you creating the plugin, to avoid having to maintain your custom snippets.\nCommon solutions My plugin does not get activated Check the profile and see if the plugin is enabled. Check the file permissions of the plugin file itself. Additionally check the /var/log/lynis.log file for additional hints.\nI can\u0026rsquo;t program, can you help? Sure we can. Users of Lynis Enterprise Suite will get additional help, depending on your needs.\n","permalink":"https://linux-audit.com/lynis/how-to-use-lynis-plugins/","tags":["lynis","one-time"],"title":"How to use Lynis plugins"},{"categories":["Auditing","Intrusion Detection","Linux"],"contents":"The Linux audit framework logs events, as specified by the configured watches. To extract particular events we can use the ausearch or aureport tools. The latter is the one we will focus on in this article, to get the most out of the tool.\nAureport The aureport utility can be executed without any parameters. It will then extract all audit events available from the log. Since the audit log can be very big, it might be better to use the -start parameter, together with a time interval (e.g. this-month).\n# aureport -start this-month Summary Report ====================== Range of time in logs: 03/01/2014 00:00:01.801 - 03/20/2014 10:17:01.209 Selected time for report: 03/01/2014 00:00:00 - 03/20/2014 10:17:01.209 Number of changes in configuration: 61 Number of changes to accounts, groups, or roles: 0 Number of logins: 0 Number of failed logins: 0 Number of authentications: 0 Number of failed authentications: 0 Number of users: 4 Number of terminals: 7 Number of host names: 0 Number of executables: 13 Number of files: 155 Number of AVC\u0026#39;s: 0 Number of MAC events: 0 Number of failed syscalls: 94 Number of anomaly events: 2 Number of responses to anomaly events: 0 Number of crypto events: 0 Number of keys: 3 Number of process IDs: 12796 Number of events: 16648 Each individual entry shown in this list, can be extracted independently.\n# aureport -file File Report =============================================== \\# date time file syscall success exe auid event =============================================== 1. 11/07/2013 19:28:45 console 0 unset ? -1 2996 2. 11/07/2013 19:52:22 /var/run/ 2 yes /bin/dash 0 3011 3. 11/07/2013 19:53:11 /var/run/ 263 yes /bin/rm 0 3012 4. 11/07/2013 20:20:04 /var/run/ 2 yes /bin/dash 1000 3032 5. 11/07/2013 20:20:04 /var/run/ 82 yes /usr/sbin/sshd 1000 3034 Each event ID (first column) can be then analyzed by using ausearch -event \u0026lt;id\u0026gt;.\nConfiguration changes Another good item to watch for is configuration changes of the audit rules themselves. Malicious people don\u0026rsquo;t like them being traced and attacking the audit configuration is a common first target. For that same reason, we suggest to sync audit logs also to an external system, with additional safeguards to protect against compromise.\n# aureport -c Config Change Report =================================== # date time type auid success event =================================== 1. 11/07/2013 20:20:04 CONFIG_CHANGE 1000 yes 3033 2. 11/07/2013 21:50:01 CONFIG_CHANGE 1000 yes 3088 3. 11/08/2013 10:35:35 CONFIG_CHANGE 1000 yes 3520 4. 11/08/2013 10:38:39 CONFIG_CHANGE 1000 yes 3527 Different audit logs When using archived logs for example, aureport needs a hint to read that file instead. This can be done by using aureport -if or aureport -input, followed by the file.\n","permalink":"https://linux-audit.com/linux-audit-framework/linux-audit-framework-using-aureport/","tags":["audit","auditd"],"title":"Linux Audit Framework: using aureport"},{"categories":["Auditing","Compliance","Hardening","Lynis"],"contents":"Determining the level of Linux server security can only by measuring the actual implemented security safeguards. This process is called auditing and focuses on comparing common security measures with the ones implemented. While there is almost no system with all possible safeguards implemented, we still can determine how well (or badly) the system is protected.\nSecurity is about finding the weakest link(s) and associate risk with each weakness. Depending on the role of the system, sensitivity of data and possible threats, we can then select what security safeguards are appropriate. By implementing these safeguards, called hardening, we increase our security defenses. After these steps, we will compare implemented measures with our baselines to determine the level of compliance.\nAuditing To audit a Linux system we use our open source tool Lynis and perform a system scan. It runs on almost all Unix and Linux based systems and only requires a shell and root permissions. It will automatically discover the operating system, available binaries and tools to run the audit process. After that first step it will start with the first batch of tests. Each set of tests are bundled by category, so it is easy to determine on what areas additional hardening might be needed.\nAfter the scan all findings will be reported and additional information will be stored in the log files (/var/log/lynis.log). Also a hardening index will be displayed, to give the auditor a first impression on how well the system is hardened.\nHardening After running Lynis it\u0026rsquo;s time to deal with the findings. Depending on the role of the machine and the risks, it\u0026rsquo;s the auditor who should make a decision on what security controls need to be implemented. Since Lynis can\u0026rsquo;t judge this, it simply will report every possible finding.\nHardening of systems can be time consuming, so therefore each finding should be carefully analyzed. Especially production environments might stop functioning if hardening isn\u0026rsquo;t done properly.\nFor professional auditors and security professionals, the Lynis Enterprise Suite will help you with selecting the right controls. Also the right hardening snippets will be provided, so they can be tested before put in production. To help you with the implementation, a priority list is created to determine where to start. It will provide risk ratings by measuring effort and risk of each control. By combining this information and compare it with other systems, the implementation plan is customized to your environment. This enables you to select the systems which needs attention first, or determining what controls to implement to have the biggest impact on the security defenses.\nCompliance Last but not least, compliance! Auditing and hardening systems are the very first steps to improve security. To maintain the effect or earlier security efforts, it\u0026rsquo;s important to keep measuring your security level and compare them with baselines. Every system should be checked on a regular base and deviations to your standard should be detected as soon as possible. By determining the risk level of this deviations, it will be much easier to take an appropriate action or implement different security measures.\nLynis does support basic compliance checking by providing key-value pairs in the scan profile. One of these examples are file permissions or kernel security parameters. The Lynis Enterprise Suite has more possibilities to check for compliance, include defining baselines and measure the compliance rate.\n","permalink":"https://linux-audit.com/linux-server-security-three-steps-to-secure-each-system/","tags":["auditing","compliance","hardening","linux","server security"],"title":"Linux server security: Three steps to secure each system"},{"categories":["Auditing","Logging"],"contents":"The Linux kernel audit framework consists of several components including a daemon, control client, audit rules and Linux audit log. In this article we take additional measures to protect the audit.log file.\nAureport The first useful utility to parse the audit.log is aureport. Without parameters it will give a summary of all events. This includes the files, users, audit keys and also items like suspicious events (anomalies). Each sub item can be read independently by using the related parameter.\nAusearch To actually search in the audit log file, use the ausearch utility. With the -a parameter an event ID can be given, which is provided as one of the columns in the aureport output. To limit the amount of entries use the -start and/or -end parameters. When using an alternative file instead of the default /var/log/audit/audit.log , then use the -input parameter followed by the file name.\nPermissions By default the audit log is located in the /var/log/audit directory. Only root has access to this file. Since it is preferred to store this log file also on a central log host, the permissions of both the directory and log file should be adjusted to give the syslog user ID at least read access. Depending on the usage of the system the adm group could get access, or limit it to root only.\ndrwxr-x- 2 syslog adm 4096 Mar 21 05:33 audit\nRemote logging To remotely log the audit data to a central node, use the file monitor from rsyslog. Example configuration for /etc/rsyslog.conf:\n# Added (load file monitor module) $ModLoad imfile # Added (at bottom of configuration) $InputFileName /var/log/audit/audit.log $InputFileTag tag_audit_log: $InputFileStateFile audit_log $InputFileSeverity info $InputFileFacility local6 $InputRunFileMonitor local6.* @@logserver.example.com:65432 Tip: Depending on the log server software used, make sure to strip out any unwanted columns. This helps in making the files accessible again for the aureport and ausearch utilities.\n","permalink":"https://linux-audit.com/linux-audit-framework/linux-audit-log-dealing-audit-log-file/","tags":["audit","system integrity"],"title":"Linux audit log: dealing with audit.log file"},{"categories":["Hardening","Kernel","Linux"],"contents":"Every system is as strong as its weakest link. In the case of an operating system like Linux, one weakness in the kernel could result in a security breach. This article covers the Linux kernel features and how they work.\nKernel features Live kernel patching As the kernel is similar to other software, it receives updates to improve it. Now and then a security weakness is discovered in one of the subsystems of the Linux kernel. This means that Linux distributions have to create a patch, rebuild the related software packages, and distribute it. The downside is that installing a new kernel package, will not resolve the issue. After all, you are still running the same kernel that you did before applying the updates. This is what kernel patches or livepatches solve.\nA kernel patch is a specialized hook into the kernel that can alter running components. This includes changing system calls and memory allocations. It is like you are applying service and repair on a driving car. The tooling and the kernel itself have to take care that the system itself does not crash while the changes are applied. This technique has been implemented in several products with names like livepatch, kpatch, kGraft, and ksplice.\nConfiguration with sysctl To view or configure security-related parameters of the kernel, there is the /etc/sysctl.conf file. This file stores the parameters and is read during boot time. However, we can also determine the configuration during run-time, by using the sysctl tool.\nTo display all available kernel parameters:\nsysctl -a\nThis will give an extensive list of configuration settings to adjust. Also, the Linux kernel security parameters are between these items. Think of items like randomization of process IDs, up to what kind of network packets should be dropped to prevent some spoofing attacks. As can be expected, adjusting any of these parameters can actually improve the way a system is running, but also have a serious negative impact. Before adjusting any parameter, read the related documentation carefully.\nNetwork parameters Depending on the role of the machine, any system nowadays is connected via the network. When the system needs to be a web server, dealing with many concurrent connections, the network related parameters are interesting to tune. This instructs the kernel to enhance this part and for example reserve more memory, or use a more aggressive stance to drop old connections.\nAutomated scan While we could go into each and every kernel parameter, we prefer automation. Tools like Lynis also check for kernel parameters. It already has several predefined key pairs to look for and provide advice. These are configured in the scan profile and can be adjusted or extended, depending on your needs. When Lynis finds a key and it has the same value in the running configuration, it will show \u0026ldquo;OK\u0026rdquo;, else it will mark the setting as being different. Depending on the key and the function of the machine, the system administrator should carefully determine if these kernel parameters should be adjusted. If so, also the configuration file /etc/sysctl.conf should be updated, to make sure the same value is active after rebooting the system.\n","permalink":"https://linux-audit.com/kernel/linux-kernel-security-and-how-to-improve-it/","tags":["kernel","kernel hardening","linux","security","sysctl","system hardening"],"title":"Linux kernel security and how to improve it"},{"categories":["Auditing","Lynis"],"contents":"Auditing on Linux Although Unix and Linux based systems are not new, getting an extensive knowledge of the operating system takes years of practice. Even then, with all changes it might be hard to keep up, especially when being an auditor. Examples of these are the differences between package managers, the way services are started and where binaries or configuration files are located. But no worries, there is help!\nWhy Lynis? The goal of Lynis is to automate the difficulties between each system. Instead of using (outdated) benchmarks or check lists, Lynis will always be using the latest methods to extract data. In the end it is more interesting to know what packages are installed, then knowing the right commands.\nTo run Lynis only root permissions are required. Installation of the tool is possible, but not a requirement. It can be executed from a remote system or storage device (e.g. USB stick). The software is open source and freely available under the GPLv3 license. So you are assured that many people already looked at the source code. Another benefit is the great feedback the project receives from the community, to keep it properly updated.\nLynis checks available binaries and then determines what tools are available to gather the right information. Depending on each discovery it will decide to enable or disable particular tests.\nTo help you during the audit, all data of the audit process is stored in a log file (by default /var/log/lynis.log). This helps in determining what tests where performed and why some of the tests where skipped. Additionally it will log what findings there are, including any warnings or suggestions. Also particular discoveries (e.g. a path to a file) or data elements will be logged, to help in determining a follow-up plan.\nAt the end of the Lynis scan a report will be displayed. This includes the findings and a hardening index, giving a first impression on how well the system is hardened.\nWhy Lynis Enterprise Suite? Auditors, consultants and security professionals are often very flexible and might work for multiple clients. Therefore we extended Lynis with additional services, which we named the Lynis Enterprise Suite. It consists of:\nManagement interface Central reporting Customized implementation plan API to automate and integrate with other systems Flexible pricing To provide auditors with a flexible pricing plan, you can opt for using credits. Each uploaded system represents a credit. This way it\u0026rsquo;s easy to do a job for a single client, upload all data and create the related reports and implementation plan. After the audit is done, simply sweep out all data and you can use the remaining credits for the next client.\n","permalink":"https://linux-audit.com/lynis-for-auditors-linux-unix-auditing/","tags":["audit","auditing","auditor","lynis"],"title":"Lynis for Auditors: Linux and Unix auditing"},{"categories":["Auditing","Linux"],"contents":"Auditing a system can be a time-consuming job, which is no different when conducting a Linux server security audit. Within this article, we give some highlights regarding the audit and tips to automate them by using Lynis.\nThe business goal Before auditing any system, determine the business goal of the system. How critical is this system for doing business? What if the system goes down?\nUsually each system has a clear role or multiple roles, like being a web server. This also determines what users might access this system, what network communication is allowed and how data needs to be protected.\nUsers Depending on the role of the system, system administrators will always have access to the system. Besides these administrators, functional and normal users might have access to the system as well.\nFirst thing to assess is how users can access the system and what authentication back-end is being used. Most systems however, still use the local back-end (passwd and shadow file). If it\u0026rsquo;s clear how users can be authenticated, the next step is to determine who can access the system and why. Users who can access the system, but have no clear business reason for being able to do so, should be flagged.\nNetwork configuration Systems have to be connected to the network to transmit, store and process data. Proper network configuration, traffic filtering and logging should be in place.\nConfiguration Per system the network configuration should be determined. Information of interest includes the IP address, netmask, gateway, allow bridged networks and in what network segments or network zone this system is active.\nListening ports Network services open a dedicated network port to listen on. Auditing these services (e.g. with netstat -nlp) will give insights on what services are active and if they are in line with the business purpose of the system.\nFirewall To make sure only allowed connections may occur, a firewall with a default \u0026ldquo;deny all\u0026rdquo; policy could be implemented. The more critical and sensitive the data, the less amount of systems should be able to communicate with the system.\nMore: audit guide for the Linux network configuration\nSoftware packages Every system can only do its job properly with additional installed software packages. Special attention to be given to the way software is upgraded, with focus on the security updates. The upgrading of systems and software is common within company security policies, yet often it\u0026rsquo;s not properly implemented.\nSteps to audit include checking the installed packages, processes which are started during boot and active services which are available via the network.\nFile permissions Depending on the role of the system, data might be stored on the system. One important part is determining what data is stored and its sensitivity. Again with determining which users can access the system, the same has to be checked regarding the access to (sensitive) data.\nAnother area of interest are files without proper ownership, like missing their owner or related group. Also binaries with their SetUID or SetGID bit set, might be of interest to determine proper hardening of the system.\nLog files For auditing and accounting purposes, log files are usually the best location to determine what has occurred. For this same reason log files should be properly stored, protected and rotated. Start by determining if all required calls and actions are properly logged. Special focus should go to the main applications dealing with users and data.\nRemote logging is a powerful method to safeguard the alteration of log files. Malicious people or insiders might adjust logging to hide their traces. Remote logging makes the protection of valuable events (or proof) easier. Security incident and event management, or SIEM solutions can help with this goal. To check if remote logging is used, check the syslog configuration.\nMalware While malware on Linux systems is less common than other platforms, it does exist. Especially backdoors, malicious scripts and rootkits can be found in the field. To detect this kind of malware use a scanner like ClamAV, LMD, Rootkit Hunter or a commercial virus scanner.\nAutomation Manual auditing is very time consuming and prone to missing important details. Therefore we suggest to use Lynis to audit a system in a matter of minutes. After the audit it becomes quickly clear what areas need more focus, like the ones mentioned in this article.\nFor companies and auditors we suggest to use the Lynis Enterprise version, a total solution to do continuous audits. Besides the power of Lynis, it can also report on the most critical controls, provide a customized implementation plan (based on effort and risk) and do more in-depth audits with the use of additional plugins.\nUseful commands Show packages rpm -qa dpkg -l Network services ss -plant (show listening services, replacement for netstat) netstat -nlp (show listening services, for older systems) ","permalink":"https://linux-audit.com/conduct-linux-server-security-audit/","tags":["audit","file permissions","linux","security","ss"],"title":"Conducting a Linux Server Security Audit"},{"categories":["Lynis","Software"],"contents":"People wonder about the main differences between Lynis and the Lynis Enterprise version. In this article we have a look on what both products are and how you can choose between the two.\nLynis Lynis is a security auditing tool for Linux and Unix based systems. With its GPLv3 license it\u0026rsquo;s open source and freely available. The tool was first released in 2007 and has undergone a lot of development during the years. Lynis is a popular tool (1000+ downloads in just a few weeks after each release) and used by many system administrators, security professionals and auditors.\nThe focus of Lynis is performing a technical audit of Linux systems and helping the auditor discover what might be improved. To some extent it will also help by providing tips on how to solve the related findings.\nLynis Enterprise Suite As you might expect, the Lynis Enterprise version is more extended than Lynis. Actually, it is not just Lynis with some extra\u0026rsquo;s, but more a full suite. The suite itself is a solution for auditing, hardening and securing Linux and Unix based systems. It even includes Lynis as one of the core components.\nLynis Enterprise Suite consists of the following components:\nWeb based management interface Dashboard and extensive reporting Customized implementation plan (effort/risk based) Code snippets to solve findings and harden systems Lynis scanner More in-depth scans with usage of plugins With these options Lynis Enterprise is much more powerful than just Lynis. Main focus of the suite is helping companies to perform continuous audits and directly implement measures. No more waiting on your yearly security audit, but ongoing scans and improvement. When a new system is deployed lacking the required hardening, it will be discovered in a matter of a day, so it can be reviewed directly.\nAudience for Lynis Enterprise Suite\nEspecially security professionals and auditors will benefit from the additional tools the suite has to offer. Also system administrators will save a lot of time with the available code snippets and being able to perform self initiated audits. No more specialized trainings or auditing experience needed to perform a technical audit yourself!\n","permalink":"https://linux-audit.com/lynis/differences-between-lynis-and-lynis-enterprise/","tags":["lynis","software"],"title":"Differences between Lynis and Lynis Enterprise"},{"categories":["Auditing","Linux","Lynis","Software"],"contents":"There are several open source vulnerability scanners for Linux, like OpenVAS . While tools like these are powerful as well, we will have a look at Lynis, our auditing tool to detect vulnerabilities of Linux and Unix systems. Why is it different than others and how can it help you in securing your systems?\nVulnerabilities Every piece of software will have sooner or later a vulnerability, a minor or major weakness which can be abused by evildoers. Within information security we have the goal to protect the confidentiality, integrity and availability of systems and the related information (or data). One of the biggest threats to this goal are people, tools and actions which make \u0026ldquo;use\u0026rdquo; of a vulnerability. Sometimes by accident, but usually on purpose, like exploiting toolkits which search the internet for systems with a known vulnerability. Therefore it\u0026rsquo;s advised to focusing on discovering and reducing the amount of vulnerabilities as soon as possible, to prevent unauthorized people from gaining access to our systems.\nOutside versus Inside scanning Many vulnerability scanners perform on a network level (outside). They can detect missing security patches due to discovered weaknesses. Still, in many cases leaks can be present while detection via the network is close to impossible. An additional downside is version banners on which some of the tools rely, providing you with a false positive when the software vendor is using a patched version.\nLynis focuses on scanning from the inside, on the system itself. This doesn\u0026rsquo;t mean it has to be installed on the system though. Lynis can run from local or external storage and only requires root permissions. The big benefit from running it on the system itself is that all information is available, including running processes, open network ports, being able to discover user accounts etc.\nDepending on your needs and how in-depth a security scan has to be, scanning from the inside might be a preferred method. More information will be available, while the chance of getting false positives is lower as well.\n","permalink":"https://linux-audit.com/vulnerabilities/open-source-vulnerability-scanner-for-linux-systems/","tags":["linux","lynis","openvas","security","tools","vulnerabilities","vulnerability scan"],"title":"Open source vulnerability scanner for Linux systems – Lynis"},{"categories":["Auditing"],"contents":"In the world of compliance, reported break-ins on the news and many security incidents, it\u0026rsquo;s common to see a security audit showing up sooner or later. Still, many people in our field don\u0026rsquo;t like them. But what is a security audit and why should we actually embrace them?\nWhy audit? Auditing has a simple goal: check if something is configured according to best practices, a baseline or a preferred state. In an ideal situation these values are all the same and part of a properly taken decision. The reason to audit is to discover any exceptions, weaknesses and determine where there is room for improvement.\nTypes of auditing Process One of the common types of auditing is to check processes to determine weaknesses or room for improvement. Within the field of IT auditors usually check on the ITIL processes is an\nTechnical auditing The other spectrum of audits check for the technical details of systems.\nVulnerability scanning Check for weaknesses in configurations, outdated software. This type of software can reveal many weaknesses in an environment. Depending on the tool it can overwhelm system administrators, not knowing where to start fixing. A common \u0026ldquo;tool\u0026rdquo; to reduce the amount of vulnerabilities is to perform hardening and software patching.\nPenetration testing Perform similar steps as blackhat hackers to break-in, however within ethical boundaries (no destruction, leakage etc). This type of audit is usually used before big projects go live, or to retest existing environments, especially when a lot of sensitive data is stored or processed.\nNegative or positive? Since audits tends to be strict (or carefully and in-depth), the discovered items are usually less appreciated by people who take care of the related systems. Still, auditing is the process of determining what can be done and system administrators (or managers) should not feel offended by any discovered items. It\u0026rsquo;s an ongoing process to help improving the environment and definitely not about \u0026ldquo;blaming\u0026rdquo; people.\nManagers and system administrators should consider an audit as a health check of their environment. Any discovered items can be seen as something to further improve the existing environment. In some cases it can even help in gaining the required budget to finally purchase that automation tool, or vulnerability scanner and do self-scans! Instead of avoiding audits, it might be better to embrace them and actually learn something from the results.\nSecurity auditing for Linux Since this blog is about auditing the Linux and Unix platform, we can\u0026rsquo;t forget to speak about our open source auditing tool Lynis . Licensed under GPLv3, it\u0026rsquo;s free to use and runs on almost all Unix and Linux based systems. It detects vulnerabilities in system configurations, missing security patches or possible improvements in applications and its configuration files.\nLynis is an audit tool (vulnerability scanner), helping system administrators, security professionals and auditors. Proper follow-up includes hardening and retesting of system configurations and applications.\n","permalink":"https://linux-audit.com/what-is/security-audit/","tags":["audit","auditing"],"title":"What is a security audit?"},{"categories":["Auditing","FreeBSD","Hardening","Lynis"],"contents":"Lynis development has its roots on a FreeBSD system, therefore FreeBSD hardening is also easy and supported when using Lynis. People who want to audit and harden their FreeBSD system will discover Lynis to be a powerful tool for this purpose. In this article we will focus on how to audit your system with Lynis.\nPorts Lynis is available from the ports tree and usually the version is close or at the latest version. To install Lynis this way:\ncd /usr/ports/security/lynis/ \u0026amp;\u0026amp; make install clean\nor to add the package:\npkg install security/lynis\nIn case the Lynis version from the ports tree is not up-to-date, please create an issue on Bugzilla for the port. In the meantime you could download Lynis manually from the CISOfy website and extract the tarball in a temporary directory.\nRunning Lynis Running Lynis can be as simple as using the audit command to perform a scan with all tests enabled. Any tests that are not relevant for FreeBSD will be skipped.\nlynis audit system\nThe output might look something like this:\nScreenshot of security scan performed with Lynis\nNote: if you manually unpacked the tarball, use ./lynis audit system from the local directory instead.\nUsually FreeBSD installations are already pretty well hardened out of the box, as the installation requires you to install additional software. Still, it\u0026rsquo;s worth to perform an extensive audit and check the outcome of the tests. These will be displayed at the bottom of the screen, together with a hardening index and pointers to log file and report file.\nAfter the scan you are advised to have a look at the log file (default /var/log/lynis.log) to determine what has been checked in each test and any further suggestions. Warnings and suggestions will be displayed also on screen.\n","permalink":"https://linux-audit.com/freebsd-hardening-lynis/","tags":["audit","auditing","hardening","lynis"],"title":"FreeBSD hardening with Lynis"},{"categories":["Auditing"],"contents":"The Linux Audit Daemon is a framework to allow auditing events on a Linux system. Within this article we will have a look at installation, configuration and using the framework to perform Linux system and security auditing.\nAuditing goals By using a powerful audit framework, the system can track many event types to monitor and audit the system. Examples include:\nAudit file access and modification See who changed a particular file Detect unauthorized changes Monitoring of system calls and functions Detect anomalies like crashing processes Set tripwires for intrusion detection purposes Record commands used by individual users Components The framework itself has several components:\nKernel audit: hooks into the kernel to capture events and deliver them to auditd Binaries auditd: daemon to capture events and store them (log file) auditctl: client tool to configure auditd audispd: daemon to multiplex events aureport: reporting tool which reads from log file (auditd.log) ausearch: event viewer (auditd.log) autrace: using audit component in kernel to trace binaries aulast: similar to last, but instead using audit framework aulastlog: similar to lastlog, also using audit framework instead ausyscall: map syscall ID and name auvirt: displaying audit information regarding virtual machines Files audit.rules: used by auditctl to read what rules need to be used auditd.conf: configuration file of auditd Installation Debian / Ubuntu apt install auditd audispd-plugins\nFedora / Red Hat Usually already installed (package: audit and audit-libs)\nConfiguration The configuration of the audit daemon is arranged by two files, one for the daemon itself (auditd.conf) and one for the rules used by the auditctl tool (audit.rules).\nauditd.conf The file auditd.conf configures the Linux audit daemon (auditd) with focus on where and how it should log events. It also defines how to deal with full disks, log rotation and the number of logs to keep. Usually the default configuration will be appropriate for most systems.\naudit.rules To configure what events should be audited, the audit framework uses a rules file named audit.rules.\nAs with most things, use a clean start and without any loaded rules. Active rules can be determined by running auditctl with the -l parameter.\n# auditctl -l No rules In case any rules are loaded, remove them with auditctl and the -D parameter.\nTime to start with monitoring something, let\u0026rsquo;s say the /etc/passwd file. We put a \u0026lsquo;watch\u0026rsquo; on the file by defining the path and permissions to look for:\nauditctl -a exit,always -F path=/etc/passwd -F perm=wa\nBy defining the path option, we instruct the audit framework what directory or file to watch for. The permissions determine what kind of access will trigger an event. Although these look similar to file permissions, note that there is a important difference between the two. The four options are:\nr = read w = write x = execute a = attribute change Finding the related event or access to the file can be quickly traced by using the ausearch tool.\n# ausearch -f /etc/passwd time-\u0026gt;Tue Mar 18 15:17:25 2014 type=PATH msg=audit(1395152245.230:533): item=0 **name=\u0026#34;/etc/passwd\u0026#34;** inode=137627 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=NORMAL type=CWD msg=audit(1395152245.230:533): **cwd=\u0026#34;/etc/audit\u0026#34;** type=SYSCALL msg=audit(1395152245.230:533): arch=c000003e **syscall=188** success=yes exit=0 a0=d14410 a1=7f66eec38db7 a2=d4ea60 a3=1c items=1 ppid=1109 pid=4900 **auid=0** uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2 comm=\u0026#34;vi\u0026#34; **exe=\u0026#34;/bin/vi\u0026#34;** subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) Some highlights of this output are:\nThe time of the event and the name of the object, the current working path (cwd), related syscall, audit user ID (auid) and the binary (exe) performing the action upon the file. Please note that the auid defines the original user during log-in. The other user ID fields might indicate a different user, depending on the effective user being used while triggering an event.\nConverting system calls\nSyscalls are logged by an numeric value. Since there will be an overlap in these values between different architectures, the active architecture is also logged.\nBy using uname -m we can determine the architecture and use ausyscall to determine what numeric call 188 represents.\n# ausyscall x86_64 188 setxattr We now know it was a change in attribute, which makes sense as we defined our watch to trigger an event on an attribute change (perm=a).\nUsed a temporary rule and want to use the old rules again? Refresh the audit rules from a file:\nauditctl -R /etc/audit/audit.rules\nAuditing of processes under Linux Similiar to using strace, the audit framework has a tool named autrace. It uses the audit framework and adds the right rules to capture information and log it. Using ausearch the gathered information can be displayed.\nTo perform a trace, start the tool:\n# **autrace /bin/ls /tmp** autrace cannot be run with rules loaded. Please delete all rules using \u0026#39;auditctl -D\u0026#39; if you really wanted to run this command. If there are rules loaded, you see this message and you have to delete the rules first.\n# auditctl -D No rules Now we can do the trace again:\n# autrace /bin/ls /tmp Waiting to execute: /bin/ls atop.d mc-root mongodb-27017.sock suds Cleaning up... Trace complete. You can locate the records with \u0026#39;ausearch -i -p 20314\u0026#39; Display related files via ausearch:\n# ausearch --start recent -p 21023 --raw | aureport --file --summary File Summary Report =========================== total file =========================== 1 /bin/ls 1 (null) inode=1975164 dev=08:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 1 /etc/ld.so.cache 1 /lib/x86_64-linux-gnu/libselinux.so.1 1 /lib/x86_64-linux-gnu/librt.so.1 1 /lib/x86_64-linux-gnu/libacl.so.1 1 /lib/x86_64-linux-gnu/libc.so.6 1 /lib/x86_64-linux-gnu/libdl.so.2 1 /lib/x86_64-linux-gnu/libpthread.so.0 1 /lib/x86_64-linux-gnu/libattr.so.1 1 /proc/filesystems 1 /usr/lib/locale/locale-archive 1 /tmp Audit file access per user The audit framework can be used to monitor syscalls, including access to files. If you want to know what files a particular user ID accessed, use a rule like this:\nauditctl -a exit,always -F arch=x86_64 -S open -F auid=80\nAuditctl option Description -F arch=x86_64 Define what architecture is used, to monitor the right syscall (some system calls are ambiguous between architectures) -S open Select the \u0026ldquo;open\u0026rdquo; syscall -F auid=80 Define related user ID that we are interested in This kind of information is really useful for intrusion detection, but also when performing forensics on a Linux system. If you are not sure about the architecture value, use uname -m to query the right value.\nAutomation Since the Linux audit daemon can provide valuable auditing data, Lynis will check for the presence of the framework. If not available, it will advice you to install. Additionally Lynis will perform several tests to determine the log file, available rules and more.\nFor proper intrusion detection, integration with an Intrusion Detection System (IDS) is key in discover events when they occur and take appropriate actions.\nMore.. The audit daemon has more possibilities. Other examples will be listed in separated articles in the future of this blog. If you are serious about auditing the Linux platform, the Linux audit framework will definitely be a good friend!\n","permalink":"https://linux-audit.com/linux-audit-framework/configuring-and-auditing-linux-systems-with-audit-daemon/","tags":["audit","auditd","forensics","intrusion detection","linux"],"title":"Configuring and auditing Linux systems with Audit daemon"},{"categories":["Lynis"],"contents":"With every software tool receiving improvements and bug fixes, it\u0026rsquo;s important to update Lynis as well. In this article we have a look at how to easily upgrade Lynis.\nOptions Two common options to keep software up-to-date is by using a package, or the usage of a custom archive. Installing Lynis is optional, running it from remote (or local) storage is a valid option.\nLynis Packages On the CISOfy software repository you can find a Lynis package. The packages are available for systems running CentOS, Debian, Fedora, RHEL, openSUSE, Ubuntu, and others.\nFor administrators who prefer to use custom packages, it\u0026rsquo;s a good option to use the source file and build a custom package. This way the package can be installed on test servers first and then deployed to all production systems. By using a software update tool or configuration management tool (e.g. CFEngine or Puppet), new releases can be pushed and enforced.\nBuilding RPM From the authors behind Lynis there is an example spec file , if you want to build your own RPM file. Another option is the spec file that openSUSE provides.\nAfter adjusting any file paths and usually the version number, run the rpmbuild tool:\nrpmbuild -ba lynis.spec\nCustomized tarball For companies with many systems, a good alternative to a Lynis package is the usage of a personally customized tarball. Download the Lynis tarball from the website of CISOfy , extract the contents and make alterations for your auditing needs. Commons adjustments include:\nFilling in license key (scan profile) Putting plugins into the tarball Customization to Lynis After the adjustments, perform a test run on some (test) systems. When the build is stable, create a new archive and publish it on a service of your choice. One common example is the usage of a web server, which stores the latest version of the package (e.g. lynis-custom-latest.tar.gz). It can be downloaded via a daily cron job with tools like wget of CURL.\nUsing GitHub Lynis is available on GitHub . Getting the latest version is as easy as cloning the project and keep it in sync. While we suggest people to keep in sync, we also believe in testing. Therefore the previous section about customizing it, should also include testing.\nLynis Enterprise Suite users Especially users of the Enterprise version will benefit of keeping Lynis up-to-date, to receive the latest improvements and new features. Sometimes new functionality will show up in the Enterprise interface. In such case it requires Lynis to be at a certain software level to collect the related data. Additionally any bugs or suggestions reported by the community is available in the latest Lynis client.\nWe advise Enterprise users to stay up-to-date and test new releases first. Only then deploy it on the production systems. Using Lynis from software repositories may result in using an outdated version.\n","permalink":"https://linux-audit.com/lynis/how-to-update-lynis/","tags":["lynis","rpm","software"],"title":"How to update Lynis"},{"categories":["Auditing","Linux","Lynis"],"contents":"Within the field of the audit profession the usage of CAAT (Computer-assisted audit techniques) or CAATTs (computer-assisted audit tools and techniques) is growing. Lynis is filling this gap for Linux and Unix based systems. It\u0026rsquo;s a well-known and stable tool in this area and improves the audit process by automation. Only a few items could then be checked manually. This saves time, makes the audit more predictable and increases the quality of the overall audit.\nLynis Based on common shell scripting, Lynis has low requirements to run. It can extract information or directly give advice on how to improve the security defenses, by performing an in-depth security audit. Lynis will check configuration files, installed packages, check the network configuration and more. Vulnerable packages, an incorrectly configured SSH daemon or missing firewall, will be noticed and reported back to the auditor running the tool.\nHistory Lynis was created by Michael Boelen in 2007 and released as a GPLv3 licensed project. During the years it received much feedback from the open source community and many releases made the tool rock-solid.\nDevelopment efforts have been increased since last year, as part of the foundation of CISOfy. Lynis is now also an integrated part of the Lynis Enterprise suite and will continue to get updates this way. The community benefit from the releases of the open source product, while the enterprise users benefit from the many users of the tool. The latter is important to ensure stability, integrity and proper support of the tool.\nAuditors Lynis focuses on the automation of technical audits. This way it can help auditors to do their job faster, improve the quality and without having to know all the technical updates in the field.\nSome tips when performing an audit with Lynis:\nCheck for the latest version before performing an audit Create your custom scan profile and include company specific files and settings Use the -auditor option to define who performed the audit at that time Store the lynis.log and lynis-report.dat files as evidence Easy deployment Within an audit it is common to use a tool for many systems at once. Lynis is flexible for that reason and does not need to be installed. The default tar archive can be used from a temporary directory on the target system. Another option is being executed from local or remote storage (e.g. USB stick or NFS share).\nExtensions With the usage of plugins, the functionality of Lynis can be extended. Especially auditors benefit from a more extensive scan, so they can better help their clients by giving the appropriate advice. For more information of the available plugins, have a look at the CISOfy website .\nEnterprise support As there aren\u0026rsquo;t much CAATTs for Linux, it\u0026rsquo;s good to know that auditors will benefit from the flexible options within the Lynis Enterprise Suite. No expensive licenses, but a simple system with credits. Pay what you need and help your clients with their auditing needs in just a matter of hours or even minutes (full scan, set-up and reporting)!\nHappy hardening!\n","permalink":"https://linux-audit.com/caatts-for-linux-lynis/","tags":["audit","auditing","automation","linux"],"title":"CAATTs for Linux: Lynis"},{"categories":["Auditing","Hardening","Lynis"],"contents":"This article explains in a few quick steps how to start with using Lynis. A more extensive explanation can be found in the documentation of Lynis.\nOption 1: install from package Use your package manager and install the \u0026rsquo;lynis\u0026rsquo; package.\napt install lynis\nor\ndnf install lynis\nNote: typically the package repository may have an outdated version of Lynis. Consider using the official repository .\nOption 2: Download Lynis wget https://cisofy.com/files/lynis-**version**.tar.gz\nUnpack tarball tar xfvz lynis-version.tar.gz\nThis will unpack the tarball with a Lynis directory.\nRun Lynis Go to the newly created directory named lynis.\ncd lynis\nWhen running Lynis for the very first time, use the audit system command. It will start the audit process and pauses after every batch of tests.\n./lynis audit system\nAfter reading the section, press ENTER to perform the next batch. Items which show up in white, can be considered to be normal. Green usually indicates a common, preferred or safe value.\nYellow or red might indicate an unexpected result, a suggestion or serious security weakness.\nDealing with findings At the end of the Lynis scan a report is displayed with the findings, a hardening index and the location of several related files.\nThis audit overview can be used to determine what items are discovered and need more investigation. This might include serious vulnerabilities which were discovered, but also minor items. It\u0026rsquo;s even possible that some value is discovered, which is configured \u0026ldquo;weak\u0026rdquo; on purpose (e.g. depending on the role the system has).\nEach finding can be found in the log file as well. The related test ID is displayed at the end of the line. For example:\nAdd a legal banner to /etc/issue, to warn unauthorized users [test:BANN-7126] Search for the related line in the log file:\n# grep BANN-7126 /var/log/lynis.log [20:11:04] Performing test ID BANN-7126 (Check issue banner file contents) [20:11:04] Suggestion: Add a legal banner to /etc/issue, to warn unauthorized users [test:BANN-7126] While this does give us positive search results, there is more information available. Therefore it\u0026rsquo;s better to open up the log file (e.g. with less) and search for the first line matching. This will be the first line as shown in the example, as this is also the start of the test.\nThe log file might also provide hints on what has been checked and where to fix them. Still, we advise to carefully read the documentation about every configuration file or parameter.\nLynis Enterprise Suite For companies and people who need more than just vulnerability checking, there is the Lynis Enterprise Suite . This includes Lynis, management reporting, dashboards, detailed explanation on fixing vulnerabilities and improve system security. To help with automation, the enterprise version also includes snippets.\n","permalink":"https://linux-audit.com/lynis/how-to-use-lynis/","tags":["lynis","tutorial"],"title":"How to use Lynis"},{"categories":["Intrusion Detection","Linux","Malware"],"contents":"The question regarding the need for antivirus for Linux is after years still relevant. It is asked at forums and shows up regularly at Quora. As the original author of rkhunter, a malware scanner for Linux and Unix systems, I analyzed many malicious software components. You might be wondering that if there is malware, there is also a need for a scanner, right? It is actually not that easy to answer. It depends\u0026hellip;\nSo is antivirus on Linux really needed or is it simply a nice-to-have? In this article, I will give an extensive answer on how to make a well-informed decision.\nAfter reading this article, you know about:\nWhat types of malicious software there is for Linux Which antivirus solutions are available Alternative methods to secure your system Malware Malicious software or malware is a family name. It includes a wide range of software, each with their own characteristics. Most malware is still focused on the Windows platforms, followed by mobile devices. The reason is simple, it is a numbers game. The more users that are active on a platform, the great is the likelihood that a malware specimen will find new targets. Another reason is the average user on a platform. Those with less technical knowledge are also more susceptible to opening malicious attachments or getting infected on a website.\nMalware threats for Linux If we want to translate the malware threat to the Linux platform, we will have to look at the different types. Each type has a main purpose and most of them have a favorite way of propagating to other systems.\nSo let\u0026rsquo;s have a look first at the different types:\nVirus Worm Backdoor or trojan horse Rootkit Ransomware Viruses The virus is oldest of all. It was heavily used in the MS-DOS era. There were a lot of different viruses. Some did crazy things, like having the letters on the screen fall of one by one. Others were more destructive and infected every floppy disk, and after a period wipe the hard disk.\nA virus like we had in the old days is very uncommon to see now. So using an antivirus scanner for this type, is not needed on Linux.\nWorms In the period that Windows 95 was popular, a newer generation of malware was active. Like a virus, it also wants to propagate to other systems. Due to the improved network connectivity, it started using email and file shares to spread. The success of the worm was typically measured in its capability to spread and not so much in the damage itself.\nWorms definitely also hit Linux systems. For example the Lion worm in 2001, which used a buffer overflow attack on systems running BIND. This DNS server component had an issue that allowed full access to the system. The worm then prepared its host system to settle in. After it was done, it starting scanning other areas of the network to spread and repeat its tasks.\nBackdoors The next category is the backdoor or trojan horses. While not completely the same, they are very similar in that they have an unexpected back entrance. Backdoors are sometimes used for good purposes, to allow a developer to do troubleshooting. If a backdoor is not closed before shipping, it might be discovered by others.\nA trojan horse is named after the Trojan war in Greek, where a wooden horse was given as a gift. It was pulled in by the defenders of the city of Troy. During the night they found out that this was a mistake. Greek soldiers had climbed out of the wooden horse and destroyed the city together with the soldiers from outside.\nBackdoored software components are a common threat, also to Linux systems. This is especially true when combined with other forms of malware, like rootkits. In this case, a backdoored SSH daemon can be placed, which allows special entrance to the attacker to ensure access is maintained.\nRootkits This category of malware had its inception of on the Linux platform. It is named after the root user and being a toolkit. It is common to see backdoors in it, to allow the attacker to stay in control of the system. Then it provides a set of smaller tools to hide in plain sight. This can be done by replacing common system tools like ls, ps, and top.\nRansomware A fairly new generation of malware is that of taking data of users hostage and asking a ransom for it. It started on Windows systems and then spread to other platforms. Linux systems have been taken hostage as well, so this threat is real.\nSo is antivirus software for Linux needed? To give the answer on the question: It might be. The need for antivirus on Linux strongly depends on the goal of the system and the available threats. Each time you implement a security solution, it should reduce the risk by countering the related threats. For Linux systems, the purpose of the system has an important role in the decision making. So let\u0026rsquo;s have a look at some typical tasks that a Linux system can serve.\nMail server When the system is configured to receive emails and relay them, it may encounter a lot of bad emails. This includes messages that are simply spam and unwanted to emails that include a malicious payload.\nConclusion: when implementing a mail system, it definitely makes sense to use antivirus scanning capabilities to protect end-users, regardless of the operating system they might use.\nAdvice: if you are running a popular mail server daemon (MTA) like Exim, Postfix, or Sendmail, use a mail scanner plugin to filter emails on the presence of malware.\nWeb server Many of the Linux systems on the planet are used to serve some web-based application or website. If your Linux web server only shares static pages, then the risks are low. That changes if you allow file uploads and the system has multiple user accounts. Especially if these user accounts can use SSH, SFTP, and other administration protocols.\nOne of the biggest risks with web servers is web shells, typically little pieces of PHP script that allow access to the system. Another high-risk item is software that is not up-to-date, including Drupal, Joomla, WordPress, and all related plugins.\nConclusion: most web server will benefit from having a malware scanner. It depends on what type of applications are hosted. With a bigger variety of applications, the risks increase. The need for doing regular scans or even on-access scans, increases.\nAdvice: look at a combination of daily or weekly scheduled scans and combine them with on-access scanning.\nCountermeasures for malware Malware at Linux is currently still a fairly low risk. By applying the right countermeasures, you can reduce the risk even further.\nAntivirus tools for Linux ClamAV, one of the well-known antivirus tools for Linux based systems\nOne of the available options for Linux is the ClamAV scanner. This scanner can be used on the command line or perform on-access scans. ClamAV is open source software and one of the most well-known tools in this field.\nClamAV is commonly used for gateways, like scanning e-mail (attachments) or files stored on disk.\nSystem hardening Security is about protecting the weakest link in the chain. Strengthening each link up to an acceptable point, where risk, costs, and effort are balanced. Hardening is about removing unneeded parts (or links), to avoid they can be attacked. This might be reducing the number of user accounts, software components, and loadable modules. Additionally, hardening focuses on improving the remaining links in the chain. Examples include implementing a firewall, restrict access to binaries and directories.\nHardening is not just an alternative for malware scanners. It should be treated as the fundamental basis of system security. It does not make much sense to install antivirus for Linux, while at the same time allowing passwordless logins for example.\nHardening tools:\nLynis Bastille Linux OpenSCAP Software patching Most malware is using weaknesses of installed and active software components, to find a small hole and circumvent system security measures. The best way to protect a system is to focus on installing security patches as soon a possible. This minimizes the amount of time between the time a vulnerability is discovered and the time find a related exploit being used. All other updates can be installed on a regular basis on when appropriate.\nSoftware patching for Linux is together with system hardening one of the strongest and most effective methods to protect systems.\nSecurity auditing While prevention is good, it\u0026rsquo;s even better to have proper detection methods. An extensive system audit can reveal weaknesses and actually often does. These weaknesses can then be solved by the earlier mentioned hardening steps. Additionally, auditing can reveal traces of break-ins or attempts to do so. As no single security measure is flawless, regular checks should be mandatory. In case you didn\u0026rsquo;t use our tool Lynis yet, give it a try. It\u0026rsquo;s open source and freely available (GPLv3 licensed).\nStay secure!\n","permalink":"https://linux-audit.com/malware/antivirus-for-linux-really-needed/","tags":["clamav","email","linux","malware","virus"],"title":"Antivirus for Linux: is it really needed?"},{"categories":["Forensics","Incident Handling and Response","Malware"],"contents":"One day your web hoster or yourself may discover that your Linux system is slow. Upon logging in, you see a high load consumed by a suspicious process name or maybe just the Apache web server. Is your system compromised? How do you know it is? Let\u0026rsquo;s have a look at how to deal with security breaches and incident response.\nRecognizing a security breach Not all security breaches are directly visible. Attackers may have compromised your system a while ago and just planted some seeds at the time. Then finally when the resources of your system may be abused, your system could be part of a botnet.\nLooking for signs Before we dive deep into this subject of dealing with a compromised Linux system, we have to answer the biggest question: how do I know my system is compromised?\nSometimes there are signs that a clear giveaway. Examples:\nPages on your website were replaced with a \u0026ldquo;You have been hacked\u0026rdquo; text The system is missing essential binaries or they crash (segfault) after executing Unauthorized users have been created The system is hosting movies, music, or other pirated data If one of these examples apply to your system, then you know there is work to do.\nNot so obvious Unfortunately, too often the signs are not so clear. The system might misbehave and yet it is not compromised. For example, it may have been placed on a blacklist (RBL). E-mails are queuing up and web application might get stuck on delivering their email. It is also possible that the CPU is spending a lot of time on a process in trouble. For example, a disk partition is full and the process is retrying to write to disk.\nConsult a professional If you are still in doubt that a system is compromised, don\u0026rsquo;t wait and consult a security professional. Waiting might reduce the chance of securing data and gather evidence. Waiting also delays getting the system back to a healthy state.\nEvery system has a function and you should consider the consequences it might have on the integrity and confidentiality of the data stored or processed by the system. Even if the system does not have a business critical purpose, consulting with a professional will give you the best action to take. Just pulling the network or power plug isn\u0026rsquo;t always the best choice.\nBreach confirmed If you have confirmed that the Linux system is really compromised, the first step is: don\u0026rsquo;t panic.\nDon\u0026rsquo;t panic! Whatever you do from here on, stay calm and focused. It is time to take actions and to execute them according to a plan. The plan helps you to stay on the course. It is easy to take the wrong actions or overlook the consequences. Too often, a system administrator reboots the system in the hope that the issues will disappear. While this is a good approach for a healthy system, it may turn the compromised system into a total mess. All data may be gone, the provided services are down, and evidence may be destructed.\nIncident response plan Let\u0026rsquo;s assume your system is compromised. We need an action plan to deal with a security incident. While many steps will be similar to most situations, your plan might need some customization. This mainly depends on the importance of your system and the services it provides. For example, if the system provides services to your customers, then you might need to get them involved. This means most likely informing them about the intrusion and letting them know what steps they need to take.\nAction steps Assign someone to take notes While things are hectic, it is easy to forget important details. Make notes of what technical actions have been performed. Also include important decisions, especially when management is involved. Document who has given you the permission to take a particular action.\nGood note-taking will be rewarding at the moment of reflection. It helps to better understand what happened and which of the actions were good or bad. Especially if more systems are involved, it may also help in troubleshooting or even reduce the impact of the security incident.\nDraft a communication plan Decide who needs to be informed, when, and how often. Make someone responsible for the communication. Preferably this is someone from the department that deals with marketing, PR, or communication. It might also be your CEO, as he or she has the right stature to address customers, employees, and partners.\nTry to be as clear as possible in your communication. Explain what happened and what actions you have taken so far. Share the steps you expect to take next and when more updates will follow.\nInclude the security team or security officer Don\u0026rsquo;t try to solve everything alone, even if you are the most technical person in the company. A security breach might impact the whole company and your actions can be costly. Get someone involved and have them help you during the decision making process.\nInform the legal department If you have a Legal department, get them involved. A security breach might have to be reported to the authorities.\nReport to the police department or CERT The police were never very good with digital crimes or cybercrime. That is changing, especially now more specific departments are created to deal with this. So consider reporting it at your local police station or a digital equivalent of it. During resolving the issue, you might need to collect evidence, especially if you consider taking legal actions later on. Also, make sure that any actions you take are according to the law in your country.\nThink ahead\nFrom a technical point of view, it\u0026rsquo;s wise to think ahead. How are you going to rebuild the services? Is a replacement system needed, or is the current pool of machines capable of handling it? What if more systems are compromised?\nPrepare\nTaking preparation steps to make things go smoother later on, can be a very wise decision. For example, rolling out some fresh virtual machines in the background and have them at the latest update level. If the decision is made to switch over to new systems, you saved some time. Especially with the flexibility nowadays, decide where you can take some preparations upfront and save you some time.\nRebuilding the system Perform a fresh installation The first step now is the installation of the new system. With virtual machine technology, it may be fine to reuse the system. Simply detach the existing disk and add a new one. The detached disk can be used for forensics when needed.\nThis is also a good time to learn from the breach. So while you are doing a new install, choose for a \u0026ldquo;minimal installation\u0026rdquo;. Any package that is not installed, can\u0026rsquo;t become an issue.\nSecurity updates Before bringing any system into production, make sure the system is up-to-date and has all security patches. It\u0026rsquo;s a waste of time and other resources to activate a new system, finding out it\u0026rsquo;s compromised again in just a matter of hours or even minutes!\nSoftware Time to install the required software. Only installing the required software will speed up the installation process, but also keeps the system clean. This is the time to avoid clutter!\nRestore of data After the system is reinstalled, data can be restored. Additional measures might be needed to prevent restoring data with anything related to the previous breach. For example, if you would restore binaries, you might bring back backdoored system utilities. Only restore the data than you can fully trust.\nDeploy the system When everything is clean and the system is tested, it can be deployed to the production farm again. As a follow-up, it\u0026rsquo;s wise to keep this system carefully monitored. The attacker might want to break the security defenses again.\nAudit The point is to learn from every experience, including a compromised Linux system. So after installation of new software, it is time to do another round of auditing. Run Lynis on the system to check if any vulnerabilities exist and fix them before deploying the system.\nAdditional thoughts Depending on your organization, you might have to deal with forensics as well. In this case, it is important that each step you take, does not make the work of the forensics team harder (or even impossible). Before altering the system, determine if your actions are authorized and do not obstruct with any forensics at a later stage.\nCollecting evidence and data is a completely field in its own. Still, there are some great guidelines and valuable lessons, which can be found in RFC3227: Guidelines for Evidence Collection and Archiving .\n","permalink":"https://linux-audit.com/dealing-with-a-compromised-linux-system/","tags":["forensics","intrusion detection","linux","malware"],"title":"How to deal with a compromised Linux system"},{"categories":["Software"],"contents":"No system can do its job without any installed software packages. However after installation of the system, or running it for a while, it often becomes unclear why some software was ever installed. This article looks at methods on auditing installed software, check for security updates and the related follow-up.\nPackage managers To enable system administrators to properly manage software and upgrading them, Linux uses a package manager. This suite often consists of a package database, the software packages itself and several support tools. These tools in particular are used to query the database, install/remove software and assist in the upgrade process. But as usual, there are often some less known parameters which might make your job easier. For auditors it is especially interesting to know what options are available, to gather more specific information focused on proper software management.\nSince there are many Linux package managers, we will focus on YUM in this article. Others will be discussed in a separate article. Still, the general principles will apply to others as well.\nRepositories Most Linux distributions make use of a so called software repository. This list of software enables the vendor to centrally maintain software packages. For the user, it means software can quickly be installed (no fiddling with cdroms) and query the central system for available software updates.\nPackage signing To protect the centrally stored packages, it\u0026rsquo;s important that no malicious updates can placed in this central location without detection. Vendors can protect their users by signing each packet digitally and have the user tools validate the related signature before installation.\nTo enable packet signing, one should ensure that GPG checks are enabled.\n/etc/yum.conf\ngpgcheck=1\nYUM security plugin Red Hat and some of the derivatives like Scientific have the possibility to install a security plugin for YUM. This way more information can be gathered about any available security advisories or upgrades.\nDisplay available advisories and related packages (with version):\n# yum list-sec SLBA-2013:0961-1 bugfix module-init-tools-3.9-21.el6\\_4.x86\\_64 SLBA-2013:1647-1 bugfix mysql-libs-5.1.71-1.el6.x86_64 SLSA-2014:0164-1 moderate/Sec. mysql-libs-5.1.73-3.el6\\_5.x86\\_64 SLSA-2013:1861-1 moderate/Sec. nss-3.15.3-3.el6\\_5.x86\\_64 SLBA-2013:1558-1 bugfix nss-softokn-3.14.3-9.el6.x86_64 SLBA-2013:1558-1 bugfix nss-softokn-freebl-3.14.3-9.el6.x86_64 SLSA-2013:1861-1 moderate/Sec. nss-sysinit-3.15.3-3.el6\\_5.x86\\_64 SLSA-2013:1861-1 moderate/Sec. nss-tools-3.15.3-3.el6\\_5.x86\\_64 SLBA-2013:0598-1 bugfix openldap-2.4.23-32.el6\\_4.x86\\_64 SLBA-2013:0778-1 bugfix openldap-2.4.23-32.el6\\_4.1.x86\\_64 SLSA-2014:0126-1 moderate/Sec. openldap-2.4.23-34.el6\\_5.1.x86\\_64 SLSA-2014:0015-1 important/Sec. openssl-1.0.1e-16.el6\\_5.4.x86\\_64 Display available packages:\n# yum updateinfo security 2\u0026gt; /dev/null Loaded plugins: security SLSA-2013:0567-1 important/Sec. kernel-2.6.32-358.0.1.el6.x86_64 SLSA-2013:0630-1 important/Sec. kernel-2.6.32-358.2.1.el6.x86_64 SLSA-2013:0744-1 important/Sec. kernel-2.6.32-358.6.1.el6.x86_64 SLSA-2013:0830-1 important/Sec. kernel-2.6.32-358.6.2.el6.x86_64 SLSA-2013:0911-1 important/Sec. kernel-2.6.32-358.11.1.el6.x86_64 SLSA-2013:1051-1 moderate/Sec. kernel-2.6.32-358.14.1.el6.x86_64 SLSA-2013:1173-1 important/Sec. kernel-2.6.32-358.18.1.el6.x86_64 SLSA-2013:1436-1 moderate/Sec. kernel-2.6.32-358.23.2.el6.x86_64 SLSA-2013:1645-2 important/Sec. kernel-2.6.32-431.el6.x86_64 SLSA-2013:1801-1 important/Sec. kernel-2.6.32-431.1.2.el6.x86_64 SLSA-2014:0159-1 important/Sec. kernel-2.6.32-431.5.1.el6.x86_64 SLSA-2013:0567-1 important/Sec. kernel-firmware-2.6.32-358.0.1.el6.noarch SLSA-2013:0630-1 important/Sec. kernel-firmware-2.6.32-358.2.1.el6.noarch SLSA-2013:0744-1 important/Sec. kernel-firmware-2.6.32-358.6.1.el6.noarch SLSA-2013:0830-1 important/Sec. kernel-firmware-2.6.32-358.6.2.el6.noarch SLSA-2013:0911-1 important/Sec. kernel-firmware-2.6.32-358.11.1.el6.noarch SLSA-2013:1051-1 moderate/Sec. kernel-firmware-2.6.32-358.14.1.el6.noarch SLSA-2013:1173-1 important/Sec. kernel-firmware-2.6.32-358.18.1.el6.noarch SLSA-2013:1436-1 moderate/Sec. kernel-firmware-2.6.32-358.23.2.el6.noarch SLSA-2013:1645-2 important/Sec. kernel-firmware-2.6.32-431.el6.noarch SLSA-2013:1801-1 important/Sec. kernel-firmware-2.6.32-431.1.2.el6.noarch SLSA-2014:0159-1 important/Sec. kernel-firmware-2.6.32-431.5.1.el6.noarch SLSA-2014:0164-1 moderate/Sec. mysql-libs-5.1.73-3.el6\\_5.x86\\_64 SLSA-2013:1861-1 moderate/Sec. nss-3.15.3-3.el6\\_5.x86\\_64 SLSA-2013:1861-1 moderate/Sec. nss-sysinit-3.15.3-3.el6\\_5.x86\\_64 SLSA-2013:1861-1 moderate/Sec. nss-tools-3.15.3-3.el6\\_5.x86\\_64 SLSA-2014:0126-1 moderate/Sec. openldap-2.4.23-34.el6\\_5.1.x86\\_64 SLSA-2014:0015-1 important/Sec. openssl-1.0.1e-16.el6\\_5.4.x86\\_64 SLSA-2014:0151-1 low/Sec. wget-1.12-1.11.el6\\_5.x86\\_64 updateinfo list done To get a more friendly overview, use yum -security check-update\n9 package(s) needed for security, out of 82 available\nb\nkernel.x86_64 2.6.32-431.5.1.el6 sl-security\nkernel-firmware.noarch 2.6.32-431.5.1.el6 sl-security\nmysql-libs.x86_64 5.1.73-3.el6_5 sl-security\nnss.x86_64 3.15.3-3.el6_5 sl-security\nnss-sysinit.x86_64 3.15.3-3.el6_5 sl-security\nnss-tools.x86_64 3.15.3-3.el6_5 sl-security\nopenldap.x86_64 2.4.23-34.el6_5.1 sl-security\nopenssl.x86_64 1.0.1e-16.el6_5.4 sl-security\ntzdata.noarch 2013i-2.el6 sl6x-security\nwget.x86_64 1.12-1.11.el6_5 sl-security\nxorg-x11-drv-ati-firmware.noarch 7.1.0-3.el6 sl6x\nThis command also displays how many updates are specifically related to security, while also displaying the total amount of available updates. For an auditor this command output provides the proper evidence what kind of software upgrade policy is used (no patching, some patches, all patches) and the current status.\nBugzilla / CVE To display specific bugfixes based on the bugzilla ID (if present), use yum list-sec bugzillas or yum list-sec cve to get CVE ID\u0026rsquo;s.\nOutput (partial):\n920961 bugfix udev-147-2.51.el6.x86_64\n947067 bugfix udev-147-2.51.el6.x86_64\n982902 bugfix udev-147-2.51.el6.x86_64\n998237 bugfix udev-147-2.51.el6.x86_64\n967554 bugfix upstart-0.6.5-12.el6_4.1.x86_64\n950532 bugfix util-linux-ng-2.17.2-12.9.el6_4.2.x86_64\n955520 bugfix util-linux-ng-2.17.2-12.9.el6_4.3.x86_64\n816342 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n846790 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n864585 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n870128 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n870854 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n872291 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n885313 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n911756 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n915844 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n917678 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n947062 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n966735 bugfix util-linux-ng-2.17.2-12.14.el6.x86_64\n833831 low/Sec. wget-1.12-1.11.el6_5.x86_64\n795919 bugfix xorg-x11-drv-ati-firmware-7.1.0-3.el6.noarch\n822280 bugfix xorg-x11-drv-ati-firmware-7.1.0-3.el6.noarch\n879102 bugfix xorg-x11-drv-ati-firmware-7.1.0-3.el6.noarch\n882086 bugfix xorg-x11-drv-ati-firmware-7.1.0-3.el6.noarch\n907616 bugfix xorg-x11-drv-ati-firmware-7.1.0-3.el6.noarch\nInstalling only security updates For system administrators it might be a big relieve to increase patching cycles, but reduce the patches to only security related updates. This way no new features or other bug fix releases will impact the stability of production systems. By using the package manager tools and right filters as shown above, it becomes much easier to select only the security updates.\nUseful articles to continue:\nShowing Available Security Updates with DNF Missing Packages: Don\u0026rsquo;t Trust External Repositories! ","permalink":"https://linux-audit.com/auditing-linux-software-packages-managers/","tags":["audit","linux","packages","package manager","security","software","yum"],"title":"Auditing Linux: Software Packages and Managers"},{"categories":["Hardening","Lynis"],"contents":"At the end of each Lynis scan, the report will be displayed. This report will include the findings (warnings and suggestions) and general information like the number of security tests performed. Additionally, the location of the log file and report data will be displayed.\nBetween all this information there is a \u0026ldquo;Lynis hardening index\u0026rdquo; displayed. This index is unique to Lynis. The index gives the auditor an impression on how well a system is hardened. This number, however, is just an indicator on taken measures. One should not confuse it with a percentage of how \u0026ldquo;safe\u0026rdquo; a system might be.\nIncreasing the index So you want to influence the Lynis hardening index? The best way is to actually implement security safeguards! Determine what findings you have on your system and apply any measures. Hardening Lynis and Unix systems is essential to get your security inline with your security policies. Besides that, no company or administrator want their systems being the target of a break-in.\nAn alternative to increasing the Lynis hardening index is determining what tests are too strict for the role of the particular machine. These tests can then be disabled in the scan profile, resulting in the test to be skipped. By using this method, the hardening rating for those particular tests will be skipped, resulting in a different score. While this might sound like a good idea, it makes the comparison with other systems harder, unless the test is skipped for all systems.\nHardening Index++ Users of the Lynis Enterprise Suite will get an even more powerful version of the Lynis hardening index. Each system is measured and compared to other systems. Depending on the findings, a risk rating will be calculated for the individual system. Additionally, averages and a maximum score will be displayed for similar machines. This gives a better view of what systems pose the most risk or need priority in a hardening project.\n","permalink":"https://linux-audit.com/lynis/lynis-hardening-index/","tags":["lynis","one-time"],"title":"Lynis hardening index"},{"categories":["Auditing","Lynis"],"contents":"Securing a Linux system can take a lot of time. For this purpose we have written Lynis, a quick and small audit tool. It\u0026rsquo;s an open source tool and freely available. You just need root permissions and a common shell and you\u0026rsquo;re ready to do your first audit. The main audience for this tool is auditors, security professionals, penetrating testers and system administrators.\nFirst audit Most Linux distributions already have Lynis in their software repository. If not, then download Lynis and extract it in a temporary directory. Start Lynis with the lynis command, or ./lynis. Run the first scan with just the \u0026rsquo;lynis audit system\u0026rsquo; command.\nLynis will now initialize itself and determine the operating system type and check what binaries are present. After this first step is done, tests from the first category will be executed. After each section you will be asked to press Enter to continue.\nFirst time users are advised to read the text labels, especially if any warnings show up. At the end of the scan they will be summarized for your convenience. These will help you later in securing your Linux machine(s), by applying hardening measures.\nNext steps After the scan is done, the findings are listed in the scan report at the end of the screen output. Also a hardening index is displayed, giving an indication on how well the system has been secured already. Note that it\u0026rsquo;s just an informational indicator and does not tell how \u0026ldquo;safe\u0026rdquo; a system might be.\nDuring the scan much information is collected and stored in the log file (by default /var/log/lynis.log) for further analysis. For example what files were tested, what discoveries were made or what additional information is available. Consider this log file as a debug treasure chest. The report file (/var/log/lynis-report.dat) is another valuable file which contains useful audit results, including the warnings and suggestions displayed before and additional data for automatic parsing.\nSecuring Linux Linux systems can be easily secured by following each of the findings and determine if a related change is justified. For technical savvy users applying these changes might be simple, yet we do advise to be careful with making adjustments. As part of our Lynis Enterprise Suite we therefore have marked each finding in a so-called control, together with the effort needed to fix the finding and the related risk. Additionally users of the Enterprise version will get a personalized implementation plan, so they can start with hardening the right controls first. Securing Linux systems might be time consuming when not taking in account that each change should be carefully reviewed and tested.\nAutomation Depending on your needs, you could schedule Lynis and run it every week (or daily) via means of a cronjob. If you have more than 10 machines, we suggest to have a look at the Lynis Enterprise Suite. This suite will help you a lot in automating Lynis scans, collect data and properly report about all findings. Now only will be informed about what has been discovered, but also how to fix it and preferable when (priority based). Securing Linux will be easier than ever before!\nLynis plugins For people who want to do additional tests can use plugins, part of the Lynis Enterprise Suite. Besides the normal system audit, it also will be scanned for malware, possible intrusion(s) and a more in-depth scan is performed. See the other articles and the Lynis documentation for more tips.\n","permalink":"https://linux-audit.com/lynis/securing-linux-audit-lynis/","tags":["audit","linux","lynis"],"title":"Securing Linux: Audit with Lynis (an introduction into auditing)"},{"categories":["Auditing"],"contents":"Within this article we have a look on how to audit and check the network configuration of Linux and other systems. The main focus is on gathering information and discover how systems are configured. By taking these steps we will do a manual audit. For efficiency reasons we suggest to use an automated tool like Lynis.\nWhere to start? Each Linux distribution has their own way and files to configure the network. Therefore we look at the basic components needed to configure a system. Usually the most important components are:\nNetwork interfaces IP address Netmask Gateway DNS configuration Hostname The first two determine to which network segment a system belongs to. The configuration of the gateway address instructs the system on how to reach systems outside its own segment. The DNS configuration itself and the hostname, are used for resolving system names into IP addresses and back.\nNowadays the ip command is the preferred method to gather information, so we will use that as much as possible.\nNetwork interfaces Every system needs an IP address on the network to be able to communicate to other systems. On a link level there are no IP addresses involved yet. By using the ip link command we can see what links are up:\n# ip link 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:16:db:dc:f7:97 brd ff:ff:ff:ff:ff:ff 3: wlan0: \u0026lt;BROADCAST,MULTICAST\u0026gt; mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 00:1c:d3:d1:c7:e8 brd ff:ff:ff:ff:ff:ff Within this output we see a local loopback address (lo), a normal network interface card (eth0) and a wireless interface. The latter has a state of \u0026ldquo;DOWN\u0026rdquo;, meaning it\u0026rsquo;s not configured or disabled.\nTo see just the network interfaces itself, systems running systemd can use networkctl to display these.\n# networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback n/a n/a 2 enp0s3 ether n/a n/a 2 links listed. Routing When a system wants to access another system outside the local network, it will use the default gateway to find a route to its destination. Depending on the internal routing configured, only a default gateway might exist.\n# ip route default via 192.168.1.1 dev eth0 metric 100 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.20 Alternative command is route -n, which will display similar information.\nWhat to look for are the default route and any other possible routes. Systems in the same network zone should normally have a similar routing configuration. Exceptions might exist and should be investigated.\nResolving To allow a system resolving hostnames into IP addresses (and back), DNS entries have to be configured. Usually this occurs in the file /etc/resolv.conf and is done with the nameserver option.\n# cat /etc/resolv.conf nameserver 8.8.8.8 nameserver 8.8.4.4 What to look for:\nAt least two or more DNS entries configured. Test if the given DNS entries work properly Lynis can automate these tests for you.\nHostname The hostname usually stored in /etc/hostname , or in the configuration file of the related network interface. By using commands like hostname or hostnamectl, it is easy to find the hostname of the system.\nAutomation With all the differences between Linux distributions, systems like OpenBSD and FreeBSD and other Unix based systems (AIX, HP-UX, Solaris), manually auditing is time-consuming. Where possible it should be limited to a minimum and only focusing on the exceptions.\nCommon files /etc/resolv.conf /etc/network/interfaces /etc/sysconfig/network Useful commands ifconfig ip route ","permalink":"https://linux-audit.com/linux-audit-auditing-network-configuration/","tags":["audit","auditing","hostnamectl","ifconfig","linux","network"],"title":"Linux Audit: Auditing the Network Configuration"},{"categories":["Lynis"],"contents":"Normal Lynis scans take a few minutes to complete, therefore any test taking more than 1 minute, might be stuck during its test. Within this article we have a look at a few things you can do.\nWhen a particular test is taking a long time, the test might be stuck. However, that\u0026rsquo;s not always the case. To determine what Lynis is doing, open up a second terminal and start with running ps aux to see what processes are active.\nNetwork utility If host, dig or any other network utility is displayed, the particular test might be waiting for a reply. Especially if the other host is very slow or not reachable, a timeout could be 30 seconds or more, looking like Lynis is stuck. In this case waiting for a little bit more, or testing the host manually to confirm, is the best next step.\nFilesystem Another possibility is that the filesystem has many files, redirects (symbolic links) or other quirks which results in Lynis getting stuck on a file or directory listing. Most often this is seen when find or grep are visible in the process listing and stay there for some time. An additional hint can be found in the log file ( /var/log/lynis.log ), by using the tail command and see if it moves on, or keeps listing the same line.\nReporting issues In case the underlying file, process or network system has been found, it might be a quick fix to solve. If you believe that it\u0026rsquo;s a returning issue and not limited to your system only, then you are advised to report the issue.\n","permalink":"https://linux-audit.com/lynis/lynis-stuck-during-testing/","tags":["lynis","one-time","troubleshooting"],"title":"Lynis stuck during testing"},{"categories":null,"contents":"If you are interested in Linux security , this is the place for you. The internet has many resources to offer, but the high-quality content is hard to find. The focus of this blog is Linux and UNIX security. Most articles will be about auditing, system hardening, and compliance.\nGoals Our goal is to write high-quality content and make it freely available. This way you can use it to secure your systems. Missing a particular subject on the blog or found something that is relevant? Let us know!\nWho is writing? Many of the articles are written by security specialist Michael Boelen . Michael is a security developer and founder of CISOfy. He covers topics within Linux security, from system hardening to compliance and regulations.\nAbout CISOfy This blog is sponsored by CISOfy . Resources are invested into this blog to make Linux security accessible to a wider audience, from system administrators to auditors and developers.\nFeedback Your comments, ideas, and feedback are welcome! If there is a particular subject you want to see covered in an article, then contact us. Please send us an email at blog @ cisofy.com.\nAbout CISOfy Lynis Screenshot of a Unix security audit performed with Lynis.\nLynis is a battle-tested technical security audit tool . It is open source , freely available, and used by system administrators all over the world. Other users include IT auditors, security professionals like pentesters. Lynis can also be used as a client component in the Lynis Enterprise solution.\nLynis Enterprise Suite For those who maintain more than 10 systems, there is the Lynis Enterprise Suite. It is much more than just Lynis. It includes a central management interface, additional plugins for Lynis, reporting capabilities, a custom implementation guide and more helpful information. For example, it contains extended information regarding each security control. Besides that, it will show you how to solve a finding, manually or with ready-to-use hardening snippets.\n","permalink":"https://linux-audit.com/about/","tags":null,"title":"About"}]