I’m going to try to post a few howto articles for some of the simpler tasks. Many of these will be things that I use daily.
It is often necessary to read a file with bash, and act upon the entire line. There are many different ways to do this, but I’ll outline two of the simpler methods, both suitable for stacking on a single command line.
For this exercise I’ll assume the file is a list of files that we need to execute a command on.
# cat file.lst |while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #
All we’ve managed to do here is cat the file.
Alternately we could move the cat to the end of the while loop as follows:
# while read line; do echo "${line}"; done < <(cat file.lst) /tmp/file1.txt /tmp/file with space.txt #
So we have a list of txt files. If we need to rename them all to .text we could use the following command.
# cat file.lst |while read line; do newname=$(echo ${line}|sed 's/txt$/text/'); mv -v "${line}" "${newname}"; done `/tmp/file1.txt' -> `/tmp/file1.text' `/tmp/file with space.txt' -> `/tmp/file with space.text' #
I prefer to use a -v when using a loop to perform tasks such as this. Adding the -i argument (inquire) will also ask before overwriting files. Another tip when doing something potentially destructive is to put an echo before the mv so you echo the commands instead of executing them.
You also aren’t limited to getting the list of files from a file. ls, find, and many other commands can be used to obtain the list. For example:
# find /tmp -name '*.txt' | while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #
That’s it for tonight. If anyone has questions, requests, or suggestions, please post comments, or send me an email. My address is anton at this domain.
62 Comments
Thanks Anton – that helped me with a problem
You’re very welcome! It’s nice to know someone’s getting some good out of my posts.
Yep, thanks. Was useful.
Thanks Anton.
“bash read a file line by line” on google gets your page first – and it does what it says!
Thanks again.
Help me
Thanks Anton
Thanks… Helped me…
Thank you Anton,
This helped me a lot!
Thanks Anton, I am in need of this
This is very useful. Thanx
Thanks, this pointed me in the right direction for what I was trying to do.
The problem with using cat though, is any variables assigned within the while loop are local to the subprocess that bash will execute cat in, meaning that:
VALUE=”"
cat /path/to/file | while read line; do
[ "${line%%:*}" == "Keyword" ] && VALUE=”${line#*:}”
done
echo $VALUE
doesn’t work as expected (assuming there is a line ‘Keyword: something [...]‘ in /path/to/file), as VALUE is assigned local to the while loop, as bash spawned a subprocess for cat.
A solution (there’s probably others):
VALUE=”"
exec 9
A solution (there’s probably others):
VALUE=”"
exec 9
Sorry about the multiple posts.. ‘Submit Comment’ doesn’t substitute “<” for “<”..
A solution (there’s probably others):
VALUE=”"
exec 9< /path/to/file # assign file descriptor 9 to file
while read -u 9 line; do # read from file descriptor 9
[ "${line%%:*}" == "Keyword" ] && VALUE=”${line#*:}”
done
exec 9<&- # free file descriptor 9
echo $VALUE
VALUE will be assigned local to the script, and will still be set to the value of ${line#*:} after the while loop has exited. Plus, the file is read using bash builtins only
Switch the loop around like this:
Now the while loop happens in the current process and VAR is available when the loop finishes.
This is very helpful to my problem, especially using 9 descriptor.
thanks,
Xiuping
Thanks, concise and correct info.
Thanks for the info. Good stuff, this. Sloved my problem.
while read line; do
echo $line
done < file.lst
Redirecting with a < is fine if you have a file. In most cases I am using commands to generate a list.
while read line; do
echo $line
done < <(find . name '*mp3')
Or more commonly:
find . -name '*mp3'|while read line; do echo "$line" ; done
Of course, usually the list is generated by some command, but the “cat” command is unnecesary here.
A lot of the simple line processing jobs can more easily be done by use of xargs.
For example:
cat file.lst | xargs echo
Instead of:
cat file.lst |while read line; do echo “${line}”; done
Yeah, I know this page was probably just an intro on how to read line by line from a shell script, but I wanted to point out that in many cases that is not necessary.
The purpose is to show how to read a file line by line with bash and I would epxect you change the echo to a number of useful commands.
As for using xargs:
# cat file.lst
file with spaces.txt
file2.txt
#cat file.lst | xargs echo
file with spaces.txt file2.txt
#cat file.lst | while read line ; do echo $line ; done
file with spaces.txt
file2.txt
#cat file.lst | xargs rm -v
rm: cannot remove `file’: No such file or directory
rm: cannot remove `with’: No such file or directory
rm: cannot remove `spaces.txt’: No such file or directory
removed `file2.txt’
#cat file.lst|while read line ; do rm -v “$line” ; done
removed `file with spaces.txt’
removed `file2.txt’
There are differences and reasons not to use xargs, even for simple things like echo and rm.
Hi ,
Please see this example.
The input file contents are as follows:
Lets name it as test3
ABC ABC ABC ??? + —- ##
ABC ABC ABC ??? + —- ##
ABC ABC ABC ??? + —- ##
Now I am doing a while loop to read this ,like below
cat test3|while read line
do
echo $line
done
The expected output is just like the above , however i am getting the below output
ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
Could you please let me know why is this so ?
Few things about this problem
1. This problem happens only when I execute this script as a cron job , Manual execution gives the proper output
2. This works fine If I remove the “???” characters from the file test3. that is ., I am getting this problem only when the input file has the character “???”
Thanks ,
Dilip
BASH is expanding the ??? to all 3 letter files (and directories) in the current directory. Crontab runs from / so that’s what it’s matching. When I run it in my home dir I see bin, log, and src dirs.
This is the same mechanism that allows rm ??? to remove all 3 letter files.
Wrap the $line in “s to prevent bash from doing the replacement.
cat tmp.txt | while read line ; do echo “$line” ; done
That was quick simple and efficient. Thanks, Anton!
Hi,
Here all examples for “while read” are mentioned with a variable like “line”. My question is what variable should we check for the same while loop which has no variable. Example:
while read
do
here a progress bar is generated
done
I need to log the output in text file too.
your help will be appreciated greatly.
Amit
Amit,
read, without a variable, uses the default of $REPLY.
I would recommend setting the variable though just to prevent confusion when you attempt to nest a while read inside another.
Thank you Anton and everybody for messages,
i want make bash file.
everything is allright but i want get variable text other file. I can’t make this.
how can make this?
Erdem,
I’m not entirely sure I understand your question. I’m assuming that you want to place the contents of a file into a variable. You could do this by:
MYVAR=$(cat file.txt)
but i have 12 variable and i want read only one file.
Erdem,
Sorry, I don’t fully understand your question, can you try to rephrase it?
Thanks,
Anton
Anton,
— THIS IS VARIABLE FILE NAME download.txt —
prg_1_name=”Anton’s Blog”
prg_1_file=”http://……./anton.tar.gz”
prg_1_fname=”anton.tar.gz”
prg_1_dname=”anton”
etc.
.
.
i want get this file contents when bash file is execute..
how can make it?
Hey
I needed a refresher on how to read line by line from a file using a loop. I just wanted you to know that your post was extremely helpful, thank you!
Erdem,
If you have a script that’s got commands in it (like setting variables) then you can run it by using ‘.’.
Something like:
#!/bin/bash
. ./config.sh
echo $prg_1_name
damn. Because of such tricks people will never learn perl.
Thanks for a great and simple guide.
Thanks, just what I needed.
Hi Anton:
I’m usually always on windows, but today, I got a shell account from a friend, and although I knew bits and pieces about *nix and shell scripting because of being a web dev, you really helped point me in the right direction here.
Thanks,
Curtis
Hi Anton,
The piece of code on this page helped me in a lot of ways.
Thanks a lot.
Praveen
hey anton,
hope all is well, i have a question that i am sure you can answer in a heartbeat… i have a command that is returning two lines, essentially, i want to take each of those lines and make a variable out of them, how do i do that with bash?
Great help. Thanks!
I’m accessing ‘http://www.random.org/cgi-bin/checkbuf’
It returns the single-line file ‘checkbuf’ with these contents”
4%
In this case I wouldn’t want to request a number of random bytes.
How do I check the number (note it’s 4%, not 004%) and do the Right Thing?
Ted
Ted,
If you get the 4% in a var, then you could always just split in on %, take the first part, and use a regular if to handle it.
X=$(curl -s -o – http://www.random.org/cgi-bin/checkbuf)
if [ "${X%\%*}" -get 10 ] ; then
echo “X is more than 10%!”
fi
I want to read from ls -R1p for use with curl. how can i ‘filter’ the output, so that directories are ignored? i also need to prefix files in subdirectories with directory names, after stripping off the colon at the end.
I basically need to know how to look for specific characters, and how to strip them off then end.
I’d use find to get all the files.
find . -type f
This will give you a list of all files, including path.
If you need to strip off the path, use basename to do so.
> FILE=./subdir/file.txt
> FILENAME=$(basename “$FILE”)
> echo $FILENAME
file.txt
“# while read line; do echo “${line}”; done
Anton,
I have read through all of the examples on reading a file using bash, but none appear to do what I’d like. I have a text file with 10 lines of data. Each line has 1 field basically, such as 0, 10, 12, 13, 14, with the comma prepresenting a \n. I can get the bash read command to read a line with numerous fields in it, and assign those to variables, but how do I get the read command to read a field per line, assign that to a variable name, then read the next line, etc until EOL?
Thanks!
You can call read inside the loop
# cat tmp
1
2
3
4
5
6
# cat tmp | while read x1 ; do read x2 ; read x3 ; read x4 ; echo $x1 $x2 $x3 $x4 ; done
The output would be:
1 2 3 4
5 6
If you run out of file in the middle of the last loop the remaining vars will be blank.
Again, delete the previous posts… I hope is the right tag for posting those code. WordPress seems to not convert strings like <…
# while read line; do echo “${line}”; done
This is Useless Use Of Cat and very bad style. Instead, remove the cat and subshell completely and just use:
# while read line; do echo “${line}”; done
I’d also suggest not using ${line} here, because it is completely unneccessary. Quoting $line is OK. We’ll get:
# while read line; do echo “$line”; done
which will work just fine.
Uuh… same here:
# find /tmp -name ‘*.txt’ | while read line; do echo “${line}”; done
/tmp/file1.txt
/tmp/file with space.txt
#
to
# find /tmp -name ‘*.txt’ | while read line; do echo “$line”; done
/tmp/file1.txt
/tmp/file with space.txt
#
Also, this example is pure nonsense. find uses -print automagically, if you don’t specifiy and -fprint*, -print* or -exec switch. The while is completely senseless. If you want to echo the things with echo anyway and not with print, you should use -exec anyway.
# find /tmp -name ‘*.txt’ -exec echo \{\} \;
/tmp/file1.txt
/tmp/file with space.txt
#
Although I do not see any point in it, since…
# find /tmp -name ‘*.txt’
/tmp/file1.txt
/tmp/file with space.txt
#
is working just fine.
Okay, that’s all.
-Ionic
Anton,
Thank you for the response. Your example worked just great. Thank you very much! Now I need to pass the file generated by redirecting into a file, i.e., the echo $x1 $x2 $x3 $x4 ; done > ids2.txt and passing that as an argument into another bash script. Thanks again
Ionic,
The point isn’t to replace cat, or get find to output the files. That is just a simple example.
The intent is to demonstrate reading something line by line and acting upon it. echo can be replaced with any other command, or a series of commands and maybe some logic.
# find . -name ‘*jpg’ | while read pic ; do
convert ${pic} ${pic}.png
mv ${pic}.png /some/new/dir
done
Or you could pull pic apart and, replace the jpg with png properly, and maybe resize if it’s over X bytes, etc.
I like to escape my shell vars so I don’t get into problems with concatination. It’s a personal preference and while other methods work, this is what I like. That way I don’t accidently try something like “$PATHnewdir” which isn’t the same as “${PATH}newdir”
Big big help, thanks a lot.
g_m.
Thank you! That helped a lot.
@Anton’s response to Geir:
You can work around the spaces problem very easily in xargs, don’t write it off so soon.
$ cat files.lst
File with spaces.txt
file2.txt
$ cat files.lst |xargs -i rm -v “{}”
removed `File with spaces.txt’
removed `file2.txt’
$cat files.lst |xargs -i echo {}
File with spaces.txt
file2.txt
Using the find command from findutils, you can also specify -print0 to print NUL characters for spaces instead of whitespace. Xargs can them interpret the whitespace via the -0 (that’s a zero) flag. find -print0 and xargs -0 should never fail because UNIX paths won’t contain NULs.
@LightningCrash
Thanks for the tips on quoting and nulls!
Helped me fix my problem, thanks.
Nice page
I have this BASH,sed, awk, blog to share with all of you.
http://unstableme.blogspot.com/
Anton,
I have the following code to read in a file an process accordingly :
#/tmp/test.txt looks like this :
# 0-1|0-2|0-3|0-4
# 1-1||1-3|1-4
# |2-2|2-3|2-4
# 3-1|3-2||3-4
FILENAME=$1; #input filename
OIFS=$IFS;
IFS=’|';
while read col1 col2 col3 col4
do
if [ -n "$col1" ]; then
fred=”col1 = ‘$col1′, “;
else
fred=”";
fi
if [ -n "$col2" ]; then
fred=$fred”col2 = ‘$col2′, “;
else
fred=$fred;
fi
if [ -n "$col3" ]; then
fred=$fred”col3 = ‘$col3′, “;
else
fred=$fred;
fi
if [ -n "$col4" ]; then
fred=$fred”col4 = ‘$col4′”;
else
fred=$fred;
fi
echo $fred;
done <$FILENAME
#the output looks like this
col1 = ’0-1′, col2 = ’0-2′, col3 = ’0-3′, col4 = ’0-4′
col1 = ’1-1′, col3 = ’1-3′, col4 = ’1-4′
col2 = ’2-2′, col3 = ’2-3′, col4 = ’2-4′
col1 = ’3-1′, col2 = ’3-2′, col4 = ’3-4′
I want to know – can I substitute ‘col1 – col4′ for a variable, since there is about 40 fields per line on the “real” file and to do more or less the same validation repetitively seems pointless.
any suggestions would be appreciated.
thanks
In all cases above you can remove the else clause, it isn’t doing anything for you. Put ‘fred=”"‘ at the top of the loop, and just append to it in each “if”.
You can use variable variables inside the loop so you don’t have to repeat each if block.
To see how they work, try this:
VAR1=abc
abc=123
echo ${!VAR1}
Inside your while read loop you could do something like this:
fred=”"
for VAR in col1 col2 col3 col4; do
if [ -n "${!VAR} ] ; then
fred=$fred”${!VAR}”
fi
done
Thanks a lot
How to read using bash, just one single fuk**ng file!?!?
Thanks for the tip. It saved my day!
One Trackback/Pingback
[...] Thanks to the following link: http://anton.lr2.com/archives/2005/03/23/read-a-file-with-bash/ [...]
Post a Comment