If you do a lot of command line stuff in Linux, sooner or later you’ll come across the "argument list too long" error when copying, moving, deleting, listing etc files in a directory that contains a lot of files. This post looks at a couple of ways of solving this problem.
For the tests conducted in this post I created two directories, "source" and "target" and created 20,000 empty text files like so:
mkdir source mkdir target for i in `seq 1 20000`; do touch source/$i.txt; done
Doing any of these:
mv source/* target/ cp source/* target/ rm source/* ls source/*
will result in errors like this:
-bash: /bin/mv: Argument list too long -bash: /bin/cp: Argument list too long -bash: /bin/rm: Argument list too long -bash: /bin/ls: Argument list too long
To workaround the problem you can use xargs or find, or a combination of the two.
Using the "ls" command, you could do this to move all the files from source into target:
ls -1 source | xargs -i mv source/{} target/
The -i flag in xargs tells it to use string replacements and {} is the placeholder. You cannot use a pattern for ls such as "ls *.txt" because you’ll get the arugment list too long error before the data can be piped through to xargs. In order to do pattern matching you have to use the find command.
Using find to do the same move from source to target, you could do this:
find source/ -name "*.txt" -exec mv {} target ;
or this, using xargs instead of the -exec flag:
find source/ -name "*.txt" | xargs -i mv {} target/
In the "find" examples above you could omit the -name *.txt part to move everything, and/or add further pattern matches such as -type f to only match files.
All three examples above take roughly the same amount of time to run although I found in testing the find … -exec method to be slightly faster than find … xargs
The find method is better than the ls method because it allows you to specify filename patterns and can give you a lot more control over what you are copying, moving, deleting etc.
As a final set of examples, you could do the following to then delete the files from the target directory:
ls -1 target | xargs -i rm target/{}
or
find target/ -name "*.txt" -exec rm {} ;
or
find target/ -name "*.txt" | xargs -i rm {}
Having to delete/move/etc files in this way is trickier and more time consuming than being able to simply do something like rm -f source/* but sometimes this is the only way to do it if there are too many files matching your pattern.