Ubuntu File System Commands

Xargs

Read a file.txt and create a directory for each line of the file
file.txt contents = (each on a separate line) apple oranges pear
cat file.txt | sort | uniq | xargs -I {} mkdir -p /var/www/fruits/{}
find dir/ -type f -print0 | xargs -0 chmod 755
(print0 is used to make sure the null character will separate them and the -0 make sure xargs uses that null charcter
find . -name "*fruit.txt" -print0 | xargs -0 -I {} cp {} /folder/{}.backup
Find files in the current directory with fruit in the filename "{}"" is the place holder for the filename. Copy the {} to the specified folder
find . -name "*fruit.txt" -depth 1 -print0 | xargs -0 -I {} rm
find . -name "*invoice*" -print0 | xargs -0 grep -li 'outwater' | xargs -I {} cp {} /dir/{}
Find all files with the word invoice then send it to grep to search in the files for the text outwater then copy those files to the dir

Diff Command

diff -y originalfile.txt revisedfile.txt

Cut Command, Can extract contiguious text from a file. eg charcters 2 - 10 of every line

cut -c 2-10 textfile.txt
Will extract characters 2 through 10 on each line
cut -c 2-10,30-35 filename.txt
will extract 2-10 and 30-35
cut -f 2,6 -d "," filename.csv
-f along with the -d option wil allow you to add a delimiter

TR (translate Function)

replace the , in a text file with a ; then pipe it back to the file
tr ',' ';' < somefile.csv > somefile.csv

Standard Input and Standard Output

Send Sorted file to new file
sort somefile.txt > newfilename.txt
To Append use >> insted of >
Supressing Output
ls -la > /dev/null

How To Change Multiple File Extensions From The Terminal

1. Open a new terminal and create the following directory in you desktop.

cd  /home/oltjano/Desktop

mkdir unixmen_tutorial

2.  cd to unixmen_tutorial and create the following files.

a.txt   b.txt  c.txt

3.  Ok guys it is time for some action. Run the following piece of code in the terminal and see what happens.

for i in *.txt; do echo $i; done

4. The following screenshot shows the result  that you should get in your terminal.

So what we are trying to do here is running a for loop and printing every filename with the .txt in the current directory. Ok, now run the following commands. It is used to strip the extension from a file.

a=a.txt

echo ${a/.txt}

5.  Do you see the following result?

6.  Ok, now run the following piece of code in your terminal. Have the file extensions changed?

for i in *.txt;  do mv "$i" "${i/.txt}".jpg; done

Wget Command

Download a single file

$ wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.1.tar.gz

Download and store it with a different name.

$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

Download in the Background Using wget -b

For a huge download, put the download in background using wget option -b as shown below.

$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

Mask User Agent and Display wget like Browser Using wget –user-agent

Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.

 wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD

Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

$ cat > download-file-list.txt

URL1

URL2

URL3

URL4

Next, give the download-file-list.txt as argument to wget using -i option as shown below.

$ wget -i download-file-list.txt

Download a Full Website Using wget –mirror

Following is the command line which you want to execute when you want to download a full website and made available for local viewing.

$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

–mirror : turn on options suitable for mirroring.

-p : download all files that are necessary to properly display a given HTML page.

–convert-links : after the download, convert the links in document for local viewing.

-P ./LOCAL-DIR : save all the files and directories to the specified directory.

Reject Certain File Types while Downloading Using wget –reject

You have found a website which is useful, but don’t want to download the images you can specify the following.

$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED

Download Only Certain File Types Using wget -r -A

You can use this under following situations:

Download all images from a website

Download all videos from a website

Download all PDF files from a website

$ wget -r -A.pdf http://url-to-webpage-with-pdfs/

FTP Download With wget

You can use wget to perform FTP download as shown below.

Anonymous FTP download using Wget

$ wget ftp-url

FTP download using wget with username and password authentication.

$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL

Xargs Linux Command

Copy all images to external hard-drive

# ls *.jpg | xargs -n1 -i cp {} /external-hard-drive/directory

Search all jpg images in the system and archive it.

# find / -name *.jpg -type f -print | xargs tar -cvzf images.tar.gz

Download all the URLs mentioned in the url-list.txt file

# cat url-list.txt | xargs wget –c

Move Multiple folders to another directory

mv -v /home/user1/Desktop/folder1/* /var/tmp/

This will move the contents of folder1 to tmp folde

Using Grep and find to search through eml files

Using Grep and Find to search through .eml files for a specific phrase

go to the dir in question

find . -exec grep -ils 'text to find' /dev/null {} \; | xargs -I {} cp -p {} /Users/homedir/Desktop/

above will find the files and copy them to specified folder

find . -exec grep -ils 'text to find\|more text to find\|even more text' /dev/null {} \; | xargs -I {} cp -p {} /Users/homedir/Desktop/

Above will find multiple search strings

find . -type f -name ".DS_Store" -exec rm -f {} \;

Above will find Ds filese in current dir and subdir and delete them

find . -exec grep -ls 'text to find' /dev/null {} \;

find . -exec grep -H 'text to look for {} \;

find . -exec grep -n 'text to look for' /dev/null {} \;

find . -exec grep -n 'yuly' /dev/null {} \; -print >> /Volumes/RAIDset1/1share/text.txt

list files that contain the name "out"

ls -la | grep out

Find files in a dir with a string. the -i is case insensitive -w is exact word

find dirname | grep -i  string