Bash Tip of the Day! variable tricks

One very little known part of Bash is variable substitution and matching. Say you have a script where you want to use the value of a variable if it has a value, but use a default value if it is null, you can use the following example:

NEWVAL=${CURVAL:-default}

This will assign the value of ${CURVAL} to ${NEWVAL} if not null, otherwise it will assign the value of default to ${NEWVAL}. Note that this will not change the value of ${CURVAL} -- although that is possible with the following example:

NEWVAL=${CURVAL:=default}

Another great trick to use with Bash are the substitution operators. I am often trying to solve problems where I take a filename and I would like to remove a part of the name which is a consistent pattern and manipulate the filename to a new format. Logrotate often appends a dot and a number to filenames which is a great thing, but sometimes that is not desireable after the fact. You can easily change this:

export FILE=maillog.12
NEWNAME=${FILE%.*}

This might be more useful as a small script to rename rotated files to date-related files:

FILELIST=$(find /var/log/ -type f -name "*maillog*" -print)

for file in ${FILELIST}
do
  BASE=${file%.*}
  BASE=${BASE##*/}
  DATE=$(date +%Y%m%d)
  HOUR=$(tail -n 1 ${file} | awk '{print $3}' | awk -F: '{print $1$2}')
  mv ${file} /new/location/${BASE}-${DATE}-${HOUR}.log
done

And with that you have an archive location with each file named to indicate the date and time of the log.

Bash Tip of the Day! for loops

Today's tip covers Bash for loops. A simple operation that comes in handy with nearly EVERY shell script.

for i in $(seq 1 10)
do
  echo ${i}
done

Note that enclosing variables in curly braces is not required but will never let you down. Take the following code, for example:

i=1
echo $i # works
echo $i5 # won't work!
echo ${i}5 # works!

Enclosing the variable in curly braces allows you to use the variable anywhere, without fear of Bash interpreting surrounding characters as part of the variable.

SELinux Violations

Yesterday I introduced SELinux and explained how to begin using it in 'Permissive' mode without affecting any existing processes on a server. Today I'm going to cover how to interpret a violation in the logs and how to find more information which might help to resolve the issue and allow you to run in 'Enforcing' mode. Enforcing mode is where the real security comes into play and mandatory access controls are enforced.

As I showed in my previous post, a violation will show up in /var/log/messages like the following:

Dec 26 07:30:26 f7-laptop setroubleshoot: SELinux is preventing the /bin/vi from using potentially mislabeled files (ftp). For complete SELinux messages. run sealert -l 0621a8c3-b182-49cf-9116-c78a9dd52199

This message indicates that when executing 'vipw' it does not get access to my personalized vi config file (.exrc). This warning is valid since no system process should be accessing a file not labeled as a system file. To determine the context of a file, use the '-Z' flag with ls:

$ ls -Z ~/.exrc
-rw-r--r-- josh josh user_u:object_r:user_home_t /home/josh/.exrc

You can see the context is 'user_u:object_r:user_home_t'. This is clearly not a system file and that is why SELinux is preventing vipw from accessing it. To see more information about this error, run the 'sealert' command above:

sealert -l 0621a8c3-b182-49cf-9116-c78a9dd52199

Now, to include only the relevant portions, see the following:

Detailed Description
SELinux has denied /bin/vi access to potentially mislabeled file(s) (ftp).
This means that SELinux will not allow /bin/vi to use these files. It is
common for users to edit files in their home directory or tmp directories
and then move (mv) them to system directories. The problem is that the
files end up with the wrong file context which confined applications are not
allowed to access.

Allowing Access
If you want /bin/vi to access this files, you need to relabel them using
restorecon -v ftp. You might want to relabel the entire directory using
restorecon -R -v .

Additional Information

Source Context user_u:system_r:sysadm_passwd_t
Target Context user_u:object_r:user_home_t
Target Objects ftp [ dir ]
Affected RPM Packages vim-minimal-7.1.12-1.fc7 [application]
Policy RPM selinux-policy-2.6.4-46.fc7
Selinux Enabled True
Policy Type targeted
MLS Enabled True
Enforcing Mode Permissive

You can see from the above message that the process trying to access the file has a security context of 'user_u:system_r:sysadm_passwd_t' and it's fairly evident from this context that it's a system process related to authentication. Further up in the description you see the details of the violation and why it would refuse access.

If the file would have been a valid file for this process to access, and I wanted to grant that access, I could do so with the 'chcon' command:

chcon user_u:system_r:sysadm_passwd_t ~/.exrc
$ ls -Z !$
ls -Z ~/.exrc
-rw-r--r-- josh josh user_u:system_r:sysadm_passwd_t /home/josh/.exrc

This action would have failed if SELinux was in enforcing mode since it would not make sense to assign a sysadm_passwd_t context to a file in this location. This is a great example of how SELinux helps to keep your system secure without letting you mis-label files! This example also show how SELinux prevents mis-steps while giving you the opportunity to troubleshoot and find out what the real problem is.

Another great SELinux resource is the Unofficial SELinux FAQ.

SELinux for all?

I recently migrated my mail server from a Gentoo Linux host to a Fedora 7 host and decided to use SELinux to improve my security posture. This is a big deal and should provide a greater level of security for my mail server, provided I can configure the security policies correctly.

SELinux was developed with the NSA and released under the GPL to provide a method of Mandatory Access Control for Linux.

The great benefit of using SELinux is that processes are prevented from accessing files that are not defined as belonging to, or associated with that process. This is a great thing! If a daemon or process were to be compromised, it would be restricted to the specific files and resources configured in the SELinux policy. Any violations are logged by the setroubleshoot daemon in the /var/log/messages syslog file and allow the user to review in order to allow or continue denial of the access attempt(s).

Enabling SELinux is very simple upon installation of Fedora or Red Hat Linux and is prompted during install. I do not recommend enabling on an OS that is in production and would only recommend enabling SELinux on a recent install or non-production host until you are very comfortable with it and have tuned your policies according to your production operation.

To get your feet wet, enable SELinux in permissive mode. This mode will allow all processes to run as if SELinux were off, but it will log any violations via auditd in syslog to allow you to tune your policies. You are prompted for this method upon install, or post-install, you could configure this via the following method:

1. use the setenforce command: setenforce permissive

usage: setenforce [ Enforcing | Permissive | 1 | 0 ]


2. edit the /etc/selinux/config file:

SELINUX=permissive

Also, be sure to remove any mention of selinux from the /boot/grub/grub.conf file if you enable SELinux, or provide the proper flags to enable it. This is also the best way to disable SELinux if you are so inclined. An example would be:

#/boot/grub/grub.conf
title Fedora 7 (2.6.22.1-41.fc7)
root (hd0,0)
kernel /vmlinuz-2.6.22.1-41.fc7 ro root=/dev/vg01/f7_root rhgb quiet selinux=0
initrd /initrd-2.6.22.1-41.fc7.img

Note that on the kernel line I have passed 'selinux=0' to the kernel which would then disable this feature.

After enabling SELinux, check syslog frequently (/var/log/messages) to look for policy violations and tune your system. A typical policy violation would look like the following:

Dec 26 08:12:50 f7-laptop setroubleshoot: SELinux is preventing the /bin/vi from using potentially mislabeled files (.exrc). For complete SELinux messages. run sealert -l 27ae2a46-19bb-47a8-b127-068a587e9494

In the above log entry, I used 'vipw' to access the /etc/passwd file and SELinux did not allow the 'vipw' binary to access my '.exrc' file (user customized file for VIM settings).

In short, SELinux can provide a GREAT measure of security to a seasoned administrator who is willing to take the time to learn how to use it. Use permissive mode and tune the policies to fit your particular environment. I do not know of anybody who uses SELinux in production or in a business due to the issues it seems to cause by lack of training and/or expertise in the configuration phase. Remember to disable SELinux and re-create the issue before bothering the application maintainer about any issues you may have.

Linux Driver Support

One of the biggest complaints of Linux users is a lack of hardware support. While Linux supports more hardware than any other operating system, it doesn't support all of the hardware that Windows supports since manufacturers create drivers for Windows by default.

A new project created by Greg Kroah-Hartman aims to create Linux drivers for vendors -- free of charge! This is revolutionary and should pave the way for increased driver support in all areas of the hardware world. It seems that there is a great demand for this service, and great participation by developers as well.

Linux in the New York Times

There is a great article in the technology section of the New York Times which talks about Linux and Ubuntu specifically. This is great coverage!

Upgrading Mediawiki from 1.6.5 to 1.9.2

I have been running Mediawiki 1.6.5 on a Gentoo server for quite some time. Today I decided that I would perform the upgrade to 1.9.2 -- which has been installed for a while without being configured for use.

The first step that I took was to update my httpd.conf to point to the new wiki location:

#Alias /mediawiki "/usr/share/webapps/mediawiki/1.6.5/htdocs"
Alias /mediawiki "/usr/share/webapps/mediawiki/1.9.2/htdocs"

The next step was to test it out. If I had a large user base or any outside users, I would have enabled the second one without disabling the first and locked my database. I dont have any users other than myself, so I did not worry about this step.

I navigated to the URL of my installation and I received an error about the config directory not being writable. I realized here that I needed to copy by LocalSettings.php from the old installation to the new version.

cp /usr/share/webapps/mediawiki/1.6.5/htdocs/LocalSettings.php /usr/share/webapps/mediawiki/1.9.2/htdocs

After copying the settings file, I visited the URL again and this time I was able to see the content. Now on to the testing phase.

To test my new wiki version, I first navigated around to make sure I could see all of the content -- this worked flawlessly. The next test was to add some content.

After adding some content, I received an internal error that suggested that I add the following line to my LocalSettings.php:

$wgShowExceptionDetails = true;

I added the above line to the end of my LocalSettings.php and I was rewarded with the error message below:

Utf8Case.ser is missing, please run "make" in the serialized directory

Backtrace:

#0 /var/www/fuag.com/htdocs/mediawiki/languages/Language.php(972): Language::getCaseMaps()
#1 /var/www/fuag.com/htdocs/mediawiki/includes/SearchUpdate.php(51): Language->stripForSearch('This will be an...')
#2 /var/www/fuag.com/htdocs/mediawiki/includes/Wiki.php(295): SearchUpdate->doUpdate()
#3 /var/www/fuag.com/htdocs/mediawiki/includes/Wiki.php(277): MediaWiki->doUpdates(Array)
#4 /var/www/fuag.com/htdocs/mediawiki/index.php(49): MediaWiki->finalCleanup(Array, Object(LoadBalancer), Object(OutputPage))
#5 {main}

This was not very clear to me, as there was not serialize directory in my installation. I did some looking around and came up with the following fix, specific to Gentoo:

1. Download new mediawiki from: http://download.wikimedia.org/mediawiki/1.9/mediawiki-1.9.3.tar.gz

curl -C - -O 'http://download.wikimedia.org/mediawiki/1.9/mediawiki-1.9.3.tar.gz'

2. Un-tar in /tmp and copy the serialized directory from the downloaded version to my local copy:

tar xzf mediawiki-1.9.3.tar.gz

cp -a serialized/ /usr/share/webapps/mediawiki/1.9.2/htdocs/

3. Run make in the new directory

make

4. Change permissions on the new directory to match the rest of my stuff in htdocs:

chown -R root:web_devs serialized/

5. Restart apache and it should now work!

Don't forget to comment out the added line in LocalSettings.php so that any end-users don't see your specific error messages.

Now I can remove all of the in-between versions that I have not been using.

primary mediawiki # ll
total 32
drwxrwsr-x 6 root web_devs 4096 Mar  4 15:14 1.6.5
drwxr-xr-x 7 root web_devs 4096 May 23  2006 1.6.6
drwxr-xr-x 7 root web_devs 4096 Jun  8  2006 1.6.7
drwxr-xr-x 7 root web_devs 4096 Sep  3 09:13 1.6.8
drwxr-xr-x 7 root web_devs 4096 Nov 28 23:45 1.7.1
drwxr-xr-x 7 root web_devs 4096 Jan 13 21:15 1.8.2
drwxr-xr-x 7 root web_devs 4096 Jan 19 22:10 1.9.0
drwxr-xr-x 6 root web_devs 4096 Mar  4 16:07 1.9.2
primary mediawiki #

Fedora Core 7 -- Going free?

Fedora Core is one of the most popular free distributions of Linux and is the community version of Redhat Enterprise Linux.

With the pending release of Fedora Core 7, the Fedora project is attempting to merge the two primary repositories; Fedora Core and Fedora Extras. With the merging of these two repositories, the Fedora project is also considering requiring all packages in the repository to be "Free Software". (re: link.)

This change would benefit everybody involved with the Fedora Project. Users of Fedora would know that license restrictions were similar throughout the software available and developers would have an easier time integrating existing software with other packages provided.

Fedora has had a lot of negativity lately around packages not working well together and updates breaking fully functional systems. This change for Fedora Core 7 might improve much of the mess that has been made.

rootkit defense -- not so fast

A recent demonstration at Black Hat DC by Joanna Rutkowska shows how hardware based rootkit defense may not be sufficient when facing an elite hacker. Rutkowska showed how the image of volatile memory as seen by the CPU can be different from the image in memory.

This is significant as it shows that monitoring at the hardware level is not fool-proof and defenders must use a combination of hardware and software monitoring.

Read the article.

Actual implementation of this attack does not seem very likely due to the complexity.

Fun with OpenLDAP, nscd, Cyrus-Imap, and Bind

After updating a server this weekend, I restarted to verify that all services would come back up and that's where the fun started...

The first issue I noticed was that Apache would not start. Nothing, I would issue a restart command:

primary ~ # /etc/init.d/apache2 restart

and I get no output, just a new prompt. The only problem here is that nothing is listening on port 80 or 443 and there are no processes running owned by apache.

primary ~ # netstat -anp --inet | egrep '80|443'

primary ~ # ps -eaf | grep apache

Nothing...

From here, I notice that bind isn't running and my email is getting queued up in postfix and not being delivered to my Cyrus-IMAP server. I see a lot of messages like this:

master[17957]: process 25160 exited, signaled to death by 13

...and...

Mar  2 14:53:43 primary master[15457]: process 15228 exited, signaled to death by 13
Mar  2 14:53:43 primary master[15457]: service imaps pid 15228 in READY state: terminated abnormally
Mar  2 14:53:43 primary master[15457]: process 15229 exited, signaled to death by 13
Mar  2 14:53:43 primary master[15457]: service imaps pid 15229 in READY state: terminated abnormally
Mar  2 14:53:43 primary master[15457]: process 15230 exited, signaled to death by 13

...and...

Mar 2 14:53:43 primary master[15457]: service lmtpunix pid 15318 in READY state: terminated abnormally
Mar 2 14:53:43 primary master[15457]: process 15319 exited, signaled to death by 13
Mar 2 14:53:43 primary master[15457]: service imaps pid 15319 in READY state: terminated abnormally
Mar 2 14:53:43 primary master[15457]: process 15320 exited, signaled to death by 13

At this point, I'm at a loss as to what is wrong and I start checking config files -- although this doesn't make a lot of sense since I didn't change any config files...(I do it anyway).

After a few hours of troubleshooting, I've re-installed Apache, Cyrus, Bind and tried different versions of each. I ended up getting Bind to work after compiling it without thread support, although this is not a good thing, as it means something else is wrong. I'm getting pretty frustrated without a working IMAP server and beginning to reconsider my decision to go with Cyrus in the first place.

Well...to make a long story short, it turns out that a few days prior to the updates, I had disabled nscd on this host to see how my OpenLDAP server would stand up to the increased load resulting from no caching. This did not seem to cause any problems and I had made this change persistent. Anyway, after searching around on the web for a while I discovered that another person had a similar issue with Cyrus where it would not start and it was due to an LDAP server being unavailable.

It turns out that if nscd is not available when Cyrus starts, and a multi-threaded bind, and Apache with my configuration, and the system is using LDAP for authentication, then these applications will not startup and run normally.

Big LESSON LEARNED for me -- always document changes, even at home on home servers, and make sure you restart the system after making big changes to verify that they don't affect other daemons. I learn this lesson about once a month on a new server, maybe this time it will persist.