Search and replace with Vim and Sed

Using search and replace is a great way to save time when editing large files in Linx. Becoming proficient with this task will increase your efficiency and will reduce your time spent doing tedious and error-prone file edits by hand.

For the sake of this tutorial, I’m going to use a copy of the /etc/apt/sources.list file to illustrate some of the changes that we can make. From your users home directory copy the sources.list file to your home directory. Or you can copy the contents of my listed below into a new file.

This tutorial assumes that you are comfortable reading and writing files with vim. If not open a terminal and type:  vimtutor

sudo cp /etc/apt/sources.list /home/luke/sources.txt
sudo chown $USER:$USER sources.txt

Here are the contents of my sources.list file.

cat sources.txt
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial main restricted
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates main restricted
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial universe
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial multiverse
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial multiverse
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates multiverse
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ xenial-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu xenial-security main restricted
deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted
deb http://security.ubuntu.com/ubuntu xenial-security universe
deb-src http://security.ubuntu.com/ubuntu xenial-security universe
deb http://security.ubuntu.com/ubuntu xenial-security multiverse
deb-src http://security.ubuntu.com/ubuntu xenial-security multiverse

 Vim

In vim, you can use the :substitute command, which is pretty much always abbreviated as just  :s  followed by a pattern that is separated by forward slashes like this.  :s/find/replace/ . This command tells vim to search the current line for “find” and replace it with with “replace”.

To perform a search of all lines change  :s to  :%s . The % in front of “s” tells Vim that we want to search every line and will replace the first instance of our search term with our replace term as this example will show.

vim sources.txt
:%s/xenial/yakkety/

Pressing enter will show give you a summary of how many changes are being made. You can review the file for accuracy and then save the file or quit without saving. Notice that every line is searched.

Vim global search and replace

To change every instance of a word and not just the first instance on a line we need to add the global option to our command.

vim sources.txt
:%s/ubuntu/OLD YELLER/g

Here you can see that every instance of “ubuntu” has been changed to “OLD YELLER”.

If you wanted to find and delete a word you can use the format  %s/<search term>// leaving the replace field empty. For example to remove comment’s from the file.

:$s/#//

Sed – Stream Editor

The command syntax for  sed actually isn’t much different from vim. By default  sed will output to standard output and will not make changes to a file unless you specify options to allow it to save changes.

Similar to our vim example to change every occurrence of “xenial” to “yakkety”:

sed 's/xenial/yakkety/' sources.txt

 sed global edit

As with vim, you can use the “g” option to make global changes. By default  sed will only change the first instance of a search term unless you append g to the end of the search line.

sed 's/ubuntu/OLD YELLER/g' sources.txt

Deleting words from a file works the same way as vim as well.

sed 's/ubuntu//g' sources.txt

 

sed with in-place editing

You can make changes to files with  sed-i . Using the -i option will create a backup copy of the original file before making edits, which will save you from inadvertently ruining a working configuration.

sed -i.bak 's/xenial/yakkety/' sources.txt

Learning to use search and replace in sed and vim will make your life as a Linux Administrator far easier and is well worth the time you will spend becoming comfortable with using these tools.

Command not found!

So you’re running through some instructions to configure software on your system, or troubleshoot some problem with a service and you see an error at the command line that says “command not found”. Here is how to locate the packages you need to install in order to use commands that are not available on your system.

CentOS/Red Hat – yum provides

Yum is an excellent package manager with lots of great built in functions. Using  yum provides <command> will output a list of packages that provide the command you are trying to run. Here is an example of the output.

sudo yum provides vgscan
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: repo1.sea.innoscale.net
 * epel: mirror.cogentco.com
 * extras: mirror.cisp.com
 * nux-dextop: mirror.li.nux.ro
 * updates: ftp.linux.ncsu.edu
7:lvm2-2.02.166-1.el7.x86_64 : Userland logical volume management tools
Repo        : base
Matched from:
Filename    : /usr/sbin/vgscan



7:lvm2-2.02.166-1.el7_3.1.x86_64 : Userland logical volume management tools
Repo        : updates
Matched from:
Filename    : /usr/sbin/vgscan

Another good thing about  yum provides is that it will also search for files. For example if you have a file on your system that you would like to match to a specific package or service  yum can get that information for you. For example you might not be sure which package installed the file /etc/sysconfig/authconfig  yum provides can get that information for you.

sudo yum provides /etc/sysconfig/authconfig
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: repo1.sea.innoscale.net
 * epel: mirror.cogentco.com
 * extras: mirror.cisp.com
 * nux-dextop: mirror.li.nux.ro
 * updates: ftp.linux.ncsu.edu
authconfig-6.2.8-14.el7.x86_64 : Command line tool for setting up authentication from network
                               : services
Repo        : base
Matched from:
Filename    : /etc/sysconfig/authconfig



authconfig-6.2.8-14.el7.x86_64 : Command line tool for setting up authentication from network
                               : services
Repo        : installed
Matched from:
Filename    : /etc/sysconfig/authconfig



authconfig-6.2.8-10.el7.x86_64 : Command line tool for setting up authentication from network
                               : services
Repo        : @base
Matched from:
Filename    : /etc/sysconfig/authconfig

 


Ubuntu

With Ubuntu 14.04 and up you don’t need to run a special command to find a program. For instance if you try to run the command  sar without having first installed sysstat you will see the following message:

luke@test-srv01:~$ sar
The program 'sar' can be found in the following packages:
 * sysstat
 * atsar
Try: sudo apt-get install <selected package>

It even tells you how to install the packages you need at the end of the message. Assuming you read the error messages you get when something doesn’t work…. Some of us may or may not be guilty of neglecting to pay attention to error messages.

OpenSUSE/Suse Enterprise Linux – cnf

Similar to Ubuntu running a command that doesn’t exist on your system will provide a suggestion to find the command you need.

luke@test-srv02:~> sar
If 'sar' is not a typo you can use command-not-found to lookup the package that contains it, like this:
    cnf sar

OpenSUSE suggests that we run another command (cnf) to find our package.

luke@test-srv02:~> cnf sar
                   
The program 'sar' can be found in the following package:
  * sysstat [ path: /usr/bin/sar, repository: zypp (SMT-http_smt-ec2_susecloud_net:SLES12-SP2-Pool) ]

Try installing with:
    sudo zypper install sysstat

Suse like Ubuntu gives us a suggestion to install sysstat and even provides the full command to get it. A simple copy and paste should be enough to get the package you want and get back to work.

What to do when df and du report different usage.

You may occasionally come across an issue where running df will produce output that disagree’s with the output of the du command. If you aren’t familiar with these two commands do see my post about filesystem and directory size. The reason for the difference in reported size is that df does not differentiate between files that are open in memory but have been deleted, or altered on the disk, whereas du will only see the files that are on the disk. You should recognize that these tools serve different functions and that you will need to rely on both of them to get a truly accurate portrayal of disk usage on your system.

Lets say you run  df -h to get an idea of how much space you have on each of the filesystems on your server or PC only to see that /var is 98% full, 9.8G out of 10G just to keep it simple. Like a good admin you run  du -h --max-depth=1 /var to find out which directories are the largest and may have files that need to be zipped up, moved, or deleted. The problem becomes apparent when  du returns that just 3G are in use on that filesystem. What do you do now?

Check for deleted files in memory.

Have you heard the old saying around the Unix world that “Everything is a file”. Well it’s true, everything in Unix, and by association Linux, is a file. This includes deleted files that now live as chunks of memory that are in use by a process.



You can view all open files on a system with the  lsof command, including deleted files that live in memory and are in use by a process (for example an old configuration file). For instance:

sudo lsof | grep root

will show you a full output of all the files currently in use by the root user. (Probably a lot of files). Running  sudo lsof | less will show you all of the open files on your system. It will look something like this. (I’m only grabbing the first 3 lines for brevity).

COMMAND     PID   TID             USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
systemd       1                   root  cwd       DIR              202,1     4096          2 /
systemd       1                   root  rtd       DIR              202,1     4096          2 /
systemd       1                   root  txt       REG              202,1  1577232     396000 /lib/systemd/systemd

Here you can see the command, the process id (PID), which user has the file open, the file descriptor (FD), the size in bytes, and the location. In our scenario we want to find out if there are any large files open that may have been deleted. We can find those files like this:

sudo lsof | grep -i deleted

Keep an eye on the 8th column which if you recall is the SIZE column. Once you identify your large files check which user has the file open (4th column), usually this will be a service account like www-data, apache, mysql. Or pay attention to the command column to identify the process or service that is using the old file. After you identify the offending process all you need to do is restart the service using  systemctl, service, or kill -HUP

In conclusion

Don’t panic, take a breath, and assess what you are seeing, think about how your tools work and what they are showing you. Above all don’t just start deleting things to free up space! The reason that df and  du are having a disagreement here is that  df see’s these deleted files along with their replacements and calculates the total disk usage,  du on the other hand only see’s the new file. Now that you know how to find the zombie files you shouldn’t have too much trouble bringing these two system tools back into agreement.

Managing packages with zypper

Suse Enterprise Linux and its free derivative OpenSUSE use a package manager called “zypper”. Zypper is a fast easy to use package management tool. In my opinion zypper is hands down the best package manager out there. It provides meaningful, easy to read output, it resolves package dependencies quickly, and it has a cool name.

Zypper has many options that can be abbreviated; install (in), search (se), info (if), update (up) and lots of others.

Basic package operations

Search for a package

Search with zypper search <search term> or zypper se <search term>

luke@OpenSUSE-tst:~> sudo zypper search apache2
Loading repository data...
Reading installed packages...
S | Name                       | Summary                                                              | Type   
--+----------------------------+----------------------------------------------------------------------+--------
  | apache2                    | The Apache Web Server Version 2.4                                    | package
  | apache2-devel              | Apache 2 Header and Include Files                                    | package
  | apache2-doc                | Additional Package Documentation                                     | package
  | apache2-event              | Apache 2 event MPM (Multi-Processing Module)                         | package
  | apache2-example-pages      | Example Pages for the Apache 2 Web Server                            | package
  | apache2-icons-oxygen       | Oxygen icons for Apache 2                                            | package
  | apache2-mod_apparmor       | AppArmor module for apache2                                          | package
  | apache2-mod_auth_kerb      | Kerberos Module for Apache                                           | package
  | apache2-mod_authn_otp      | Apache module for one-time password authentication                   | package
  | apache2-mod_dnssd          | Apache2 module for Zeroconf support via DNS-SD                       | package
  | apache2-mod_evasive        | Denial of Service evasion module for Apache                          | package
  | apache2-mod_fastcgi        | A FastCGI Module for Apache 2                                        | package
  | apache2-mod_fcgid          | Alternative FastCGI module for Apache2                               | package
  | apache2-mod_jk             | Connectors between Apache and Tomcat Servlet Container               | package
  | apache2-mod_mono           | Run ASP.NET Pages on Unix with Apache and Mono                       | package
  | apache2-mod_nss            | SSL/TLS module for the Apache HTTP server                            | package
  | apache2-mod_perl           | Embedded Perl for Apache                                             | package
  | apache2-mod_perl-devel     | Embedded Perl for Apache - Development package                       | package
  | apache2-mod_php5           | PHP5 Module for Apache 2.0                                           | package
  | apache2-mod_php7           | PHP7 Module for Apache 2.0                                           | package
  | apache2-mod_proxy_uwsgi    | uWSGI Proxy Module for Apache 2.0                                    | package
  | apache2-mod_python         | A Python Module for the Apache 2 Web Server                          | package
  | apache2-mod_scgi           | Apache module named mod_scgi that implements the client side of th-> | package
  | apache2-mod_security2      | ModSecurity Open Source Web Application Firewall                     | package
  | apache2-mod_tidy           | Apache Module using the Tidy Library to Clean Up HTML, XHTML, and -> | package
  | apache2-mod_uwsgi          | uWSGI Module for Apache 2.0                                          | package
  | apache2-mod_wsgi           | Python WSGI adapter module for Apache                                | package
  | apache2-mod_wsgi-python3   | A WSGI interface for Python3 web applications in Apache              | package
  | apache2-prefork            | Apache 2 "prefork" MPM (Multi-Processing Module)                     | package
  | apache2-utils              | Apache 2 utilities                                                   | package
  | apache2-worker             | Apache 2 worker MPM (Multi-Processing Module)                        | package
  | perl-Apache2-AuthCookieDBI | An AuthCookie module backed by a DBI database.                       | package

As you can see the output is clean and includes a short summary of each matching item in the search.

Get info on a package

Use zypper info <package> or zypper if <package>

luke@OpenSUSE-tst:~> zypper info apache2
Loading repository data...
Reading installed packages...
Information for package apache2:
--------------------------------
Repository     : Main Repository (OSS)            
Name           : apache2                          
Version        : 2.4.23-4.3                       
Arch           : x86_64                           
Vendor         : openSUSE                         
Installed Size : 4.0 MiB                          
Installed      : No                               
Status         : not installed                    
Summary        : The Apache Web Server Version 2.4
Description    :                                  
    This version of httpd is a major release of the 2.4 stable branch,
    and represents the best available version of Apache HTTP Server.
    New features include Loadable MPMs, major improvements to OCSP support,
    mod_lua, Dynamic Reverse Proxy configuration, Improved Authentication/
    Authorization, FastCGI Proxy, New Expression Parser, and a Small Object
    Caching API.
     See /usr/share/doc/packages/apache2/, http://httpd.apache.org/, and
    http://httpd.apache.org/docs-2.4/upgrading.html.

The info option will show you which repository the package is in, its name, version, whether or not it is installed, and a brief description.

Install a package

To install a package type zypper install <package name> or zypper in <package name>

luke@OpenSUSE-tst:~> sudo zypper in apache2
Loading repository data...
Reading installed packages...
Resolving package dependencies...
The following 6 NEW packages are going to be installed:
  apache2 apache2-prefork apache2-utils libapr1 libapr-util1 libnghttp2-14
6 new packages to install.
Overall download size: 1.8 MiB. Already cached: 0 B. After the operation, additional 5.4 MiB will be used.
Continue? [y/n/? shows all options] (y):

Running the install option solves for all dependencies quickly, and display’s them in a way that allows an administrator to easily see everything that will be installed and how much disk space will be consumed. In this case we see 6 new packages.

Check for updates

Check for updates with zypper list-updates or zypper lu

luke@OpenSUSE-tst:~> zypper lu
Loading repository data...
Reading installed packages...
S | Repository             | Name                        | Current Version | Available Version       | Arch  
--+------------------------+-----------------------------+-----------------+-------------------------+-------
v | Main Update Repository | ImageMagick                 | 6.8.8.1-20.1    | 6.8.8.1-22.2            | x86_64
v | Main Update Repository | alsa                        | 1.1.2-1.2       | 1.1.2-3.1               | x86_64
v | Main Update Repository | autoyast2-installation      | 3.1.151-1.2     | 3.1.154-3.1             | noarch
v | Main Update Repository | dolphin                     | 16.08.2-1.1     | 16.08.2-3.1             | x86_64
v | Main Update Repository | dolphin-part                | 16.08.2-1.1     | 16.08.2-3.1             | x86_64
v | Main Update Repository | ispell                      | 3.3.02-111.4    | 3.3.02-113.1            | x86_64
v | Main Update Repository | ispell-american             | 3.3.02-111.4    | 3.3.02-113.1            | x86_64
v | Main Update Repository | java-1_7_0-openjdk          | 1.7.0.111-35.1  | 1.7.0.121-37.2          | x86_64
v | Main Update Repository | java-1_7_0-openjdk-headless | 1.7.0.111-35.1  | 1.7.0.121-37.2          | x86_64
v | Main Update Repository | k3b                         | 2.0.3-6.21      | 2.0.3-9.1               | x86_64
v | Main Update Repository | k3b-lang                    | 2.0.3-6.21      | 2.0.3-9.1               | noarch
v | Main Update Repository | kernel-default              | 4.4.36-5.1      | 4.4.36-8.1              | x86_64
v | Main Update Repository | libMagickCore-6_Q16-1       | 6.8.8.1-20.1    | 6.8.8.1-22.2            | x86_64
v | Main Update Repository | libMagickWand-6_Q16-1       | 6.8.8.1-20.1    | 6.8.8.1-22.2            | x86_64
v | Main Update Repository | libasound2                  | 1.1.2-1.2       | 1.1.2-3.1               | x86_64
v | Main Update Repository | libblkid1                   | 2.28-5.10       | 2.28-7.1                | x86_64
v | Main Update Repository | libdolphinvcs5              | 16.08.2-1.1     | 16.08.2-3.1             | x86_64
v | Main Update Repository | libfdisk1                   | 2.28-5.10       | 2.28-7.1                | x86_64
v | Main Update Repository | libgc1                      | 7.2d-6.4        | 7.2d-8.1                | x86_64
v | Main Update Repository | libgit2-24                  | 0.24.1-1.16     | 0.24.1-3.1              | x86_64
v | Main Update Repository | libmount1                   | 2.28-5.10       | 2.28-7.1                | x86_64
v | Main Update Repository | libpcre1                    | 8.33-4.51       | 8.39-6.1                | x86_64
v | Main Update Repository | libpcre1-32bit              | 8.33-4.51       | 8.39-6.1                | x86_64
v | Main Update Repository | libpcre16-0                 | 8.33-4.51       | 8.39-6.1                | x86_64
v | Main Update Repository | libsensors4                 | 3.4.0-2.4       | 3.4.0-4.1               | x86_64
v | Main Update Repository | libsmartcols1               | 2.28-5.10       | 2.28-7.1                | x86_64
v | Main Update Repository | libsystemd0                 | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | libsystemd0-32bit           | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | libudev1                    | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | libuuid1                    | 2.28-5.10       | 2.28-7.1                | x86_64
v | Main Update Repository | libwicked-0-6               | 0.6.38-2.1      | 0.6.39-4.1              | x86_64
v | Main Update Repository | systemd                     | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | systemd-32bit               | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | systemd-bash-completion     | 228-13.1        | 228-15.1                | noarch
v | Main Update Repository | systemd-logger              | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | systemd-sysvinit            | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | udev                        | 228-13.1        | 228-15.1                | x86_64
v | Main Update Repository | util-linux                  | 2.28-5.10       | 2.28-7.1                | x86_64
v | Main Update Repository | util-linux-lang             | 2.28-5.10       | 2.28-7.1                | noarch
v | Main Update Repository | util-linux-systemd          | 2.28-5.2        | 2.28-7.1                | x86_64
v | Main Update Repository | w3m                         | 0.5.3-158.2     | 0.5.3.git20161120-160.1 | x86_64
v | Main Update Repository | wicked                      | 0.6.38-2.1      | 0.6.39-4.1              | x86_64
v | Main Update Repository | wicked-service              | 0.6.38-2.1      | 0.6.39-4.1              | x86_64
v | Main Update Repository | xdm                         | 1.1.11-11.1     | 1.1.11-13.1             | x86_64
v | Main Update Repository | xdm-xsession                | 1.1.11-11.1     | 1.1.11-13.1             | x86_64
v | Main Update Repository | yast2-services-manager      | 3.1.42-16.1     | 3.1.43-18.1             | noarch

List updates presents a table that shows which repository an update will come from, which packages have an update, the current version, and the latest available version.

Update with zypper

Update the system with zypper update or zypper up

luke@OpenSUSE-tst:~> sudo zypper up
Loading repository data...
Reading installed packages...
The following NEW package is going to be installed:
  kernel-default-4.4.36-8.1
The following application is going to be REMOVED:
  Dolphin
The following 45 packages are going to be upgraded:
  alsa autoyast2-installation dolphin dolphin-part ImageMagick ispell ispell-american java-1_7_0-openjdk
  java-1_7_0-openjdk-headless k3b k3b-lang libasound2 libblkid1 libdolphinvcs5 libfdisk1 libgc1 libgit2-24
  libMagickCore-6_Q16-1 libMagickWand-6_Q16-1 libmount1 libpcre1 libpcre1-32bit libpcre16-0 libsensors4
  libsmartcols1 libsystemd0 libsystemd0-32bit libudev1 libuuid1 libwicked-0-6 systemd systemd-32bit
  systemd-bash-completion systemd-logger systemd-sysvinit udev util-linux util-linux-lang util-linux-systemd w3m
  wicked wicked-service xdm xdm-xsession yast2-services-manager
45 packages to upgrade, 1 new.
Overall download size: 121.1 MiB. Already cached: 0 B. After the operation, additional 237.9 MiB will be used.
Continue? [y/n/? shows all options] (y):

Once again you see each package that is to be updated along with disk usage.

After the update has completed use zypper ps to show which processes, if any, need to be restarted. In this case we updated the kernel so it’s a good idea to do a full reboot.

List installed packages

To get a list of all installed packages try running

sudo zypper se -i

 Find patterns

Suse uses the term patterns to describe groups of packages that are commonly installed together. For instance a mail server will have an email service, maybe an antivirus scanner, a spam blocker…

Try sudo zypper patternsto get a list of installable patterns.

More info

For further reading you can always check the man pages man zypper or check out Suse documentation here: https://www.suse.com/documentation/sles-12/book_sle_admin/data/sec_zypper.html

Uncomplicated Firewall… be careful

If like me, you enjoy the simplicity of UFW (Uncomplicated Firewall) on your Ubuntu servers be careful when you turn it on. UFW doesn’t have any default rules to allow ssh inbound by default, if you aren’t careful when turning it on you could find yourself locked out, and if you don’t have direct console access to the server that could mean being locked out forever! Not a conversation you want to have with a client, or your boss,… or tech support at your friendly cloud provider. Continue reading “Uncomplicated Firewall… be careful”