Search and replace with Vim and Sed

Using search and replace is a great way to save time when editing large files in Linx. Becoming proficient with this task will increase your efficiency and will reduce your time spent doing tedious and error-prone file edits by hand.

For the sake of this tutorial, I’m going to use a copy of the /etc/apt/sources.list file to illustrate some of the changes that we can make. From your users home directory copy the sources.list file to your home directory. Or you can copy the contents of my listed below into a new file.

This tutorial assumes that you are comfortable reading and writing files with vim. If not open a terminal and type:  vimtutor

sudo cp /etc/apt/sources.list /home/luke/sources.txt
sudo chown $USER:$USER sources.txt

Here are the contents of my sources.list file.

cat sources.txt
# See for how to upgrade to
# newer versions of the distribution.
deb xenial main restricted
deb-src xenial main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb xenial-updates main restricted
deb-src xenial-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb xenial universe
deb-src xenial universe
deb xenial-updates universe
deb-src xenial-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb xenial multiverse
deb-src xenial multiverse
deb xenial-updates multiverse
deb-src xenial-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb xenial-backports main restricted universe multiverse
deb-src xenial-backports main restricted universe multiverse

deb xenial-security main restricted
deb-src xenial-security main restricted
deb xenial-security universe
deb-src xenial-security universe
deb xenial-security multiverse
deb-src xenial-security multiverse


In vim, you can use the :substitute command, which is pretty much always abbreviated as just  :s  followed by a pattern that is separated by forward slashes like this.  :s/find/replace/ . This command tells vim to search the current line for “find” and replace it with with “replace”.

To perform a search of all lines change  :s to  :%s . The % in front of “s” tells Vim that we want to search every line and will replace the first instance of our search term with our replace term as this example will show.

vim sources.txt

Pressing enter will show give you a summary of how many changes are being made. You can review the file for accuracy and then save the file or quit without saving. Notice that every line is searched.

Vim global search and replace

To change every instance of a word and not just the first instance on a line we need to add the global option to our command.

vim sources.txt
:%s/ubuntu/OLD YELLER/g

Here you can see that every instance of “ubuntu” has been changed to “OLD YELLER”.

If you wanted to find and delete a word you can use the format  %s/<search term>// leaving the replace field empty. For example to remove comment’s from the file.


Sed – Stream Editor

The command syntax for  sed actually isn’t much different from vim. By default  sed will output to standard output and will not make changes to a file unless you specify options to allow it to save changes.

Similar to our vim example to change every occurrence of “xenial” to “yakkety”:

sed 's/xenial/yakkety/' sources.txt

 sed global edit

As with vim, you can use the “g” option to make global changes. By default  sed will only change the first instance of a search term unless you append g to the end of the search line.

sed 's/ubuntu/OLD YELLER/g' sources.txt

Deleting words from a file works the same way as vim as well.

sed 's/ubuntu//g' sources.txt


sed with in-place editing

You can make changes to files with  sed-i . Using the -i option will create a backup copy of the original file before making edits, which will save you from inadvertently ruining a working configuration.

sed -i.bak 's/xenial/yakkety/' sources.txt

Learning to use search and replace in sed and vim will make your life as a Linux Administrator far easier and is well worth the time you will spend becoming comfortable with using these tools.

Command not found!

So you’re running through some instructions to configure software on your system, or troubleshoot some problem with a service and you see an error at the command line that says “command not found”. Here is how to locate the packages you need to install in order to use commands that are not available on your system.

CentOS/Red Hat – yum provides

Yum is an excellent package manager with lots of great built in functions. Using  yum provides <command> will output a list of packages that provide the command you are trying to run. Here is an example of the output.

sudo yum provides vgscan
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base:
 * epel:
 * extras:
 * nux-dextop:
 * updates:
7:lvm2-2.02.166-1.el7.x86_64 : Userland logical volume management tools
Repo        : base
Matched from:
Filename    : /usr/sbin/vgscan

7:lvm2-2.02.166-1.el7_3.1.x86_64 : Userland logical volume management tools
Repo        : updates
Matched from:
Filename    : /usr/sbin/vgscan

Another good thing about  yum provides is that it will also search for files. For example if you have a file on your system that you would like to match to a specific package or service  yum can get that information for you. For example you might not be sure which package installed the file /etc/sysconfig/authconfig  yum provides can get that information for you.

sudo yum provides /etc/sysconfig/authconfig
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base:
 * epel:
 * extras:
 * nux-dextop:
 * updates:
authconfig-6.2.8-14.el7.x86_64 : Command line tool for setting up authentication from network
                               : services
Repo        : base
Matched from:
Filename    : /etc/sysconfig/authconfig

authconfig-6.2.8-14.el7.x86_64 : Command line tool for setting up authentication from network
                               : services
Repo        : installed
Matched from:
Filename    : /etc/sysconfig/authconfig

authconfig-6.2.8-10.el7.x86_64 : Command line tool for setting up authentication from network
                               : services
Repo        : @base
Matched from:
Filename    : /etc/sysconfig/authconfig



With Ubuntu 14.04 and up you don’t need to run a special command to find a program. For instance if you try to run the command  sar without having first installed sysstat you will see the following message:

luke@test-srv01:~$ sar
The program 'sar' can be found in the following packages:
 * sysstat
 * atsar
Try: sudo apt-get install <selected package>

It even tells you how to install the packages you need at the end of the message. Assuming you read the error messages you get when something doesn’t work…. Some of us may or may not be guilty of neglecting to pay attention to error messages.

OpenSUSE/Suse Enterprise Linux – cnf

Similar to Ubuntu running a command that doesn’t exist on your system will provide a suggestion to find the command you need.

luke@test-srv02:~> sar
If 'sar' is not a typo you can use command-not-found to lookup the package that contains it, like this:
    cnf sar

OpenSUSE suggests that we run another command (cnf) to find our package.

luke@test-srv02:~> cnf sar
The program 'sar' can be found in the following package:
  * sysstat [ path: /usr/bin/sar, repository: zypp (SMT-http_smt-ec2_susecloud_net:SLES12-SP2-Pool) ]

Try installing with:
    sudo zypper install sysstat

Suse like Ubuntu gives us a suggestion to install sysstat and even provides the full command to get it. A simple copy and paste should be enough to get the package you want and get back to work.

What to do when df and du report different usage.

You may occasionally come across an issue where running df will produce output that disagree’s with the output of the du command. If you aren’t familiar with these two commands do see my post about filesystem and directory size. The reason for the difference in reported size is that df does not differentiate between files that are open in memory but have been deleted, or altered on the disk, whereas du will only see the files that are on the disk. You should recognize that these tools serve different functions and that you will need to rely on both of them to get a truly accurate portrayal of disk usage on your system.

Lets say you run  df -h to get an idea of how much space you have on each of the filesystems on your server or PC only to see that /var is 98% full, 9.8G out of 10G just to keep it simple. Like a good admin you run  du -h --max-depth=1 /var to find out which directories are the largest and may have files that need to be zipped up, moved, or deleted. The problem becomes apparent when  du returns that just 3G are in use on that filesystem. What do you do now?

Check for deleted files in memory.

Have you heard the old saying around the Unix world that “Everything is a file”. Well it’s true, everything in Unix, and by association Linux, is a file. This includes deleted files that now live as chunks of memory that are in use by a process.
You can view all open files on a system with the  lsof command, including deleted files that live in memory and are in use by a process (for example an old configuration file). For instance:

sudo lsof | grep root

will show you a full output of all the files currently in use by the root user. (Probably a lot of files). Running  sudo lsof | less will show you all of the open files on your system. It will look something like this. (I’m only grabbing the first 3 lines for brevity).

COMMAND     PID   TID             USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
systemd       1                   root  cwd       DIR              202,1     4096          2 /
systemd       1                   root  rtd       DIR              202,1     4096          2 /
systemd       1                   root  txt       REG              202,1  1577232     396000 /lib/systemd/systemd

Here you can see the command, the process id (PID), which user has the file open, the file descriptor (FD), the size in bytes, and the location. In our scenario we want to find out if there are any large files open that may have been deleted. We can find those files like this:

sudo lsof | grep -i deleted

Keep an eye on the 8th column which if you recall is the SIZE column. Once you identify your large files check which user has the file open (4th column), usually this will be a service account like www-data, apache, mysql. Or pay attention to the command column to identify the process or service that is using the old file. After you identify the offending process all you need to do is restart the service using  systemctl, service, or kill -HUP

In conclusion

Don’t panic, take a breath, and assess what you are seeing, think about how your tools work and what they are showing you. Above all don’t just start deleting things to free up space! The reason that df and  du are having a disagreement here is that  df see’s these deleted files along with their replacements and calculates the total disk usage,  du on the other hand only see’s the new file. Now that you know how to find the zombie files you shouldn’t have too much trouble bringing these two system tools back into agreement.