read a file with bash
I’m going to try to post a few howto articles for some of the simpler tasks. Many of these will be things that I use daily.
It is often necessary to read a file with bash, and act upon the entire line. There are many different ways to do this, but I’ll outline two of the simpler methods, both suitable for stacking on a single command line.
For this exercise I’ll assume the file is a list of files that we need to execute a command on.
# cat file.lst |while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #
All we’ve managed to do here is cat the file.
Alternately we could move the cat to the end of the while loop as follows:
# while read line; do echo "${line}"; done < <(cat file.lst) /tmp/file1.txt /tmp/file with space.txt #
So we have a list of txt files. If we need to rename them all to .text we could use the following command.
# cat file.lst |while read line; do newname=$(echo ${line}|sed 's/txt$/text/'); mv -v "${line}" "${newname}"; done `/tmp/file1.txt' -> `/tmp/file1.text' `/tmp/file with space.txt' -> `/tmp/file with space.text' #
I prefer to use a -v when using a loop to perform tasks such as this. Adding the -i argument (inquire) will also ask before overwriting files. Another tip when doing something potentially destructive is to put an echo before the mv so you echo the commands instead of executing them.
You also aren’t limited to getting the list of files from a file. ls, find, and many other commands can be used to obtain the list. For example:
# find /tmp -name '*.txt' | while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #
That’s it for tonight. If anyone has questions, requests, or suggestions, please post comments, or send me an email. My address is anton at this domain.
April 22nd, 2005 at 5:27 pm
Thanks Anton - that helped me with a problem
April 22nd, 2005 at 5:31 pm
You’re very welcome! It’s nice to know someone’s getting some good out of my posts.
November 18th, 2005 at 12:12 am
Yep, thanks. Was useful.
December 19th, 2005 at 7:14 am
Thanks Anton.
“bash read a file line by line” on google gets your page first - and it does what it says!
Thanks again.
January 20th, 2006 at 3:43 pm
Help me
January 20th, 2006 at 3:44 pm
Thanks Anton
February 15th, 2006 at 4:52 am
Thanks… Helped me…
February 21st, 2006 at 12:09 pm
Thank you Anton,
This helped me a lot!
March 22nd, 2006 at 1:13 pm
Thanks Anton, I am in need of this
March 27th, 2006 at 12:29 pm
[...] Thanks to the following link: http://anton.lr2.com/archives/2005/03/23/read-a-file-with-bash/ [...]
April 6th, 2006 at 12:04 pm
This is very useful. Thanx
April 10th, 2006 at 3:34 am
Thanks, this pointed me in the right direction for what I was trying to do.
The problem with using cat though, is any variables assigned within the while loop are local to the subprocess that bash will execute cat in, meaning that:
VALUE=”"
cat /path/to/file | while read line; do
[ "${line%%:*}" == "Keyword" ] && VALUE=”${line#*:}”
done
echo $VALUE
doesn’t work as expected (assuming there is a line ‘Keyword: something [...]‘ in /path/to/file), as VALUE is assigned local to the while loop, as bash spawned a subprocess for cat.
A solution (there’s probably others):
VALUE=”"
exec 9
April 10th, 2006 at 3:35 am
A solution (there’s probably others):
VALUE=”"
exec 9
April 10th, 2006 at 3:39 am
Sorry about the multiple posts.. ‘Submit Comment’ doesn’t substitute “<” for “<”..
A solution (there’s probably others):
VALUE=”"
exec 9< /path/to/file # assign file descriptor 9 to file
while read -u 9 line; do # read from file descriptor 9
[ "${line%%:*}" == "Keyword" ] && VALUE=”${line#*:}”
done
exec 9<&- # free file descriptor 9
echo $VALUE
VALUE will be assigned local to the script, and will still be set to the value of ${line#*:} after the while loop has exited. Plus, the file is read using bash builtins only
April 10th, 2006 at 4:19 pm
Switch the loop around like this:
Now the while loop happens in the current process and VAR is available when the loop finishes.
April 14th, 2006 at 8:25 pm
This is very helpful to my problem, especially using 9 descriptor.
thanks,
Xiuping
May 1st, 2006 at 6:03 pm
Thanks, concise and correct info.
May 17th, 2006 at 4:02 pm
Thanks for the info. Good stuff, this. Sloved my problem.
June 1st, 2006 at 6:48 pm
while read line; do
echo $line
done < file.lst
June 1st, 2006 at 7:27 pm
Redirecting with a < is fine if you have a file. In most cases I am using commands to generate a list.
while read line; do
echo $line
done < <(find . name '*mp3')
Or more commonly:
find . -name '*mp3'|while read line; do echo "$line" ; done
June 2nd, 2006 at 3:30 am
Of course, usually the list is generated by some command, but the “cat” command is unnecesary here.
July 12th, 2006 at 4:19 pm
A lot of the simple line processing jobs can more easily be done by use of xargs.
For example:
cat file.lst | xargs echo
Instead of:
cat file.lst |while read line; do echo “${line}”; done
Yeah, I know this page was probably just an intro on how to read line by line from a shell script, but I wanted to point out that in many cases that is not necessary.
July 12th, 2006 at 5:22 pm
The purpose is to show how to read a file line by line with bash and I would epxect you change the echo to a number of useful commands.
As for using xargs:
# cat file.lst
file with spaces.txt
file2.txt
#cat file.lst | xargs echo
file with spaces.txt file2.txt
#cat file.lst | while read line ; do echo $line ; done
file with spaces.txt
file2.txt
#cat file.lst | xargs rm -v
rm: cannot remove `file’: No such file or directory
rm: cannot remove `with’: No such file or directory
rm: cannot remove `spaces.txt’: No such file or directory
removed `file2.txt’
#cat file.lst|while read line ; do rm -v “$line” ; done
removed `file with spaces.txt’
removed `file2.txt’
There are differences and reasons not to use xargs, even for simple things like echo and rm.
July 27th, 2006 at 9:58 am
Hi ,
Please see this example.
The input file contents are as follows:
Lets name it as test3
ABC ABC ABC ??? + —- ##
ABC ABC ABC ??? + —- ##
ABC ABC ABC ??? + —- ##
Now I am doing a while loop to read this ,like below
cat test3|while read line
do
echo $line
done
The expected output is just like the above , however i am getting the below output
ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
Could you please let me know why is this so ?
Few things about this problem
1. This problem happens only when I execute this script as a cron job , Manual execution gives the proper output
2. This works fine If I remove the “???” characters from the file test3. that is ., I am getting this problem only when the input file has the character “???”
Thanks ,
Dilip
July 27th, 2006 at 10:55 am
BASH is expanding the ??? to all 3 letter files (and directories) in the current directory. Crontab runs from / so that’s what it’s matching. When I run it in my home dir I see bin, log, and src dirs.
This is the same mechanism that allows rm ??? to remove all 3 letter files.
Wrap the $line in “s to prevent bash from doing the replacement.
cat tmp.txt | while read line ; do echo “$line” ; done
August 25th, 2006 at 3:26 am
That was quick simple and efficient. Thanks, Anton!
September 11th, 2006 at 7:26 pm
Hi,
Here all examples for “while read” are mentioned with a variable like “line”. My question is what variable should we check for the same while loop which has no variable. Example:
while read
do
here a progress bar is generated
done
I need to log the output in text file too.
your help will be appreciated greatly.
Amit
September 12th, 2006 at 3:50 pm
Amit,
read, without a variable, uses the default of $REPLY.
I would recommend setting the variable though just to prevent confusion when you attempt to nest a while read inside another.
September 26th, 2006 at 11:21 am
Thank you Anton and everybody for messages,
i want make bash file.
everything is allright but i want get variable text other file. I can’t make this.
how can make this?
September 26th, 2006 at 11:32 am
Erdem,
I’m not entirely sure I understand your question. I’m assuming that you want to place the contents of a file into a variable. You could do this by:
MYVAR=$(cat file.txt)
September 26th, 2006 at 12:04 pm
but i have 12 variable and i want read only one file.
September 26th, 2006 at 12:11 pm
Erdem,
Sorry, I don’t fully understand your question, can you try to rephrase it?
Thanks,
Anton
September 26th, 2006 at 12:25 pm
Anton,
— THIS IS VARIABLE FILE NAME download.txt —
prg_1_name=”Anton’s Blog”
prg_1_file=”http://……./anton.tar.gz”
prg_1_fname=”anton.tar.gz”
prg_1_dname=”anton”
etc.
.
.
i want get this file contents when bash file is execute..
how can make it?
October 8th, 2006 at 6:47 pm
Erdem,
If you have a script that’s got commands in it (like setting variables) then you can run it by using ‘.’.
Something like:
#!/bin/bash
. ./config.sh
echo $prg_1_name
November 5th, 2006 at 10:31 pm
Hey
I needed a refresher on how to read line by line from a file using a loop. I just wanted you to know that your post was extremely helpful, thank you!
November 30th, 2006 at 3:25 am
damn. Because of such tricks people will never learn perl.
December 22nd, 2006 at 9:02 am
Thanks for a great and simple guide.
January 22nd, 2007 at 7:00 am
Thanks, just what I needed.
January 23rd, 2007 at 9:10 am
Hi Anton:
I’m usually always on windows, but today, I got a shell account from a friend, and although I knew bits and pieces about *nix and shell scripting because of being a web dev, you really helped point me in the right direction here.
Thanks,
Curtis
February 1st, 2007 at 3:52 am
Hi Anton,
The piece of code on this page helped me in a lot of ways.
Thanks a lot.
Praveen
February 21st, 2007 at 7:23 pm
hey anton,
hope all is well, i have a question that i am sure you can answer in a heartbeat… i have a command that is returning two lines, essentially, i want to take each of those lines and make a variable out of them, how do i do that with bash?
February 23rd, 2007 at 1:32 am
Great help. Thanks!
February 23rd, 2007 at 6:40 pm
I’m accessing ‘http://www.random.org/cgi-bin/checkbuf’
It returns the single-line file ‘checkbuf’ with these contents”
4%
In this case I wouldn’t want to request a number of random bytes.
How do I check the number (note it’s 4%, not 004%) and do the Right Thing?
Ted
March 1st, 2007 at 3:25 pm
Ted,
If you get the 4% in a var, then you could always just split in on %, take the first part, and use a regular if to handle it.
X=$(curl -s -o - http://www.random.org/cgi-bin/checkbuf)
if [ "${X%\%*}" -get 10 ] ; then
echo “X is more than 10%!”
fi
April 27th, 2007 at 1:51 pm
I want to read from ls -R1p for use with curl. how can i ‘filter’ the output, so that directories are ignored? i also need to prefix files in subdirectories with directory names, after stripping off the colon at the end.
I basically need to know how to look for specific characters, and how to strip them off then end.
May 1st, 2007 at 8:56 am
I’d use find to get all the files.
find . -type f
This will give you a list of all files, including path.
If you need to strip off the path, use basename to do so.
> FILE=./subdir/file.txt
> FILENAME=$(basename “$FILE”)
> echo $FILENAME
file.txt
August 22nd, 2007 at 11:36 am
“# while read line; do echo “${line}”; done
August 22nd, 2007 at 3:04 pm
Anton,
I have read through all of the examples on reading a file using bash, but none appear to do what I’d like. I have a text file with 10 lines of data. Each line has 1 field basically, such as 0, 10, 12, 13, 14, with the comma prepresenting a \n. I can get the bash read command to read a line with numerous fields in it, and assign those to variables, but how do I get the read command to read a field per line, assign that to a variable name, then read the next line, etc until EOL?
Thanks!
August 23rd, 2007 at 10:13 am
You can call read inside the loop
# cat tmp
1
2
3
4
5
6
# cat tmp | while read x1 ; do read x2 ; read x3 ; read x4 ; echo $x1 $x2 $x3 $x4 ; done
The output would be:
1 2 3 4
5 6
If you run out of file in the middle of the last loop the remaining vars will be blank.
August 24th, 2007 at 6:08 am
Again, delete the previous posts… I hope is the right tag for posting those code. Wordpress seems to not convert strings like <…
# while read line; do echo “${line}”; done
This is Useless Use Of Cat and very bad style. Instead, remove the cat and subshell completely and just use:
# while read line; do echo “${line}”; done
I’d also suggest not using ${line} here, because it is completely unneccessary. Quoting $line is OK. We’ll get:
# while read line; do echo “$line”; done
which will work just fine.
Uuh… same here:
# find /tmp -name ‘*.txt’ | while read line; do echo “${line}”; done
/tmp/file1.txt
/tmp/file with space.txt
#
to
# find /tmp -name ‘*.txt’ | while read line; do echo “$line”; done
/tmp/file1.txt
/tmp/file with space.txt
#
Also, this example is pure nonsense. find uses -print automagically, if you don’t specifiy and -fprint*, -print* or -exec switch. The while is completely senseless. If you want to echo the things with echo anyway and not with print, you should use -exec anyway.
# find /tmp -name ‘*.txt’ -exec echo \{\} \;
/tmp/file1.txt
/tmp/file with space.txt
#
Although I do not see any point in it, since…
# find /tmp -name ‘*.txt’
/tmp/file1.txt
/tmp/file with space.txt
#
is working just fine.
Okay, that’s all.
-Ionic
August 24th, 2007 at 6:58 am
Anton,
Thank you for the response. Your example worked just great. Thank you very much! Now I need to pass the file generated by redirecting into a file, i.e., the echo $x1 $x2 $x3 $x4 ; done > ids2.txt and passing that as an argument into another bash script. Thanks again
August 24th, 2007 at 10:46 am
Ionic,
The point isn’t to replace cat, or get find to output the files. That is just a simple example.
The intent is to demonstrate reading something line by line and acting upon it. echo can be replaced with any other command, or a series of commands and maybe some logic.
# find . -name ‘*jpg’ | while read pic ; do
convert ${pic} ${pic}.png
mv ${pic}.png /some/new/dir
done
Or you could pull pic apart and, replace the jpg with png properly, and maybe resize if it’s over X bytes, etc.
I like to escape my shell vars so I don’t get into problems with concatination. It’s a personal preference and while other methods work, this is what I like. That way I don’t accidently try something like “$PATHnewdir” which isn’t the same as “${PATH}newdir”
August 31st, 2007 at 3:35 am
Big big help, thanks a lot.
g_m.
September 26th, 2007 at 4:26 pm
Thank you! That helped a lot.
October 5th, 2007 at 8:17 am
@Anton’s response to Geir:
You can work around the spaces problem very easily in xargs, don’t write it off so soon.
$ cat files.lst
File with spaces.txt
file2.txt
$ cat files.lst |xargs -i rm -v “{}”
removed `File with spaces.txt’
removed `file2.txt’
$cat files.lst |xargs -i echo {}
File with spaces.txt
file2.txt
Using the find command from findutils, you can also specify -print0 to print NUL characters for spaces instead of whitespace. Xargs can them interpret the whitespace via the -0 (that’s a zero) flag. find -print0 and xargs -0 should never fail because UNIX paths won’t contain NULs.
October 5th, 2007 at 4:33 pm
@LightningCrash
Thanks for the tips on quoting and nulls!
February 15th, 2008 at 7:17 pm
Helped me fix my problem, thanks.
February 29th, 2008 at 12:28 pm
Nice page
I have this BASH,sed, awk, blog to share with all of you.
http://unstableme.blogspot.com/
September 2nd, 2008 at 4:20 am
Anton,
I have the following code to read in a file an process accordingly :
#/tmp/test.txt looks like this :
# 0-1|0-2|0-3|0-4
# 1-1||1-3|1-4
# |2-2|2-3|2-4
# 3-1|3-2||3-4
FILENAME=$1; #input filename
OIFS=$IFS;
IFS=’|';
while read col1 col2 col3 col4
do
if [ -n "$col1" ]; then
fred=”col1 = ‘$col1′, “;
else
fred=”";
fi
if [ -n "$col2" ]; then
fred=$fred”col2 = ‘$col2′, “;
else
fred=$fred;
fi
if [ -n "$col3" ]; then
fred=$fred”col3 = ‘$col3′, “;
else
fred=$fred;
fi
if [ -n "$col4" ]; then
fred=$fred”col4 = ‘$col4′”;
else
fred=$fred;
fi
echo $fred;
done <$FILENAME
#the output looks like this
col1 = ‘0-1′, col2 = ‘0-2′, col3 = ‘0-3′, col4 = ‘0-4′
col1 = ‘1-1′, col3 = ‘1-3′, col4 = ‘1-4′
col2 = ‘2-2′, col3 = ‘2-3′, col4 = ‘2-4′
col1 = ‘3-1′, col2 = ‘3-2′, col4 = ‘3-4′
I want to know - can I substitute ‘col1 - col4′ for a variable, since there is about 40 fields per line on the “real” file and to do more or less the same validation repetitively seems pointless.
any suggestions would be appreciated.
thanks
September 2nd, 2008 at 8:31 am
In all cases above you can remove the else clause, it isn’t doing anything for you. Put ‘fred=”"‘ at the top of the loop, and just append to it in each “if”.
You can use variable variables inside the loop so you don’t have to repeat each if block.
To see how they work, try this:
VAR1=abc
abc=123
echo ${!VAR1}
Inside your while read loop you could do something like this:
fred=”"
for VAR in col1 col2 col3 col4; do
if [ -n "${!VAR} ] ; then
fred=$fred”${!VAR}”
fi
done