Search

Showing posts with label Hadoop fs Shell Commands. Show all posts
Showing posts with label Hadoop fs Shell Commands. Show all posts

Using the CHMOD command effectively



Can’t remember how to calculate which CHMOD command you want to use. Below you will find every possible CHMOD command and the file permissions they will set.  Along with a simple explanation, every possible combination is listed below.
The permission mode is computed by adding up the following values for the user, the file group, and for everyone else.
  • Read 4 – Allowed to read files
  • Write 2 – Allowed to write/modify files
  • eXecute1 – Read/write/delete/modify/directory

Example 1:
user
group
world
r+w
r

4+2+0
4+0+0
0+0+0
= 640
Example 2:
user
group
world
r+w+x
w+x
x
4+2+1
0+2+1
0+0+1
= 731
Now some explanations:
Mode
Str Perms
Explanation
0477
-r–rwxrwx
owner has read only (4), other and group has rwx (7)
0677
-rw-rwxrwx
owner has rw only(6), other and group has rwx (7)
0444
-r–r–r–
all have read only (4)
0666
-rw-rw-rw-
all have rw only (6)
0400
-r——–
owner has read only(4), group and others have no permission(0)
0600
-rw——-
owner has rw only, group and others have no permission
0470
-r–rwx—
owner has read only, group has rwx, others have no permission
0407
-r—–rwx
owner has read only, other has rwx, group has no permission
0670
-rw-rwx—
owner has rw only, group has rwx, others have no permission
0607
-rw—-rwx
owner has rw only, group has no permission and others have rwx

hadoop fs -touchz

The hadoop touchz command creates a zero byte file. This is similar to the touch command in unix.

The syntax is shown below:

hadoop fs -touchz /user/DPhadoop/filename

hadoop fs -text

The hadoop text command displays the source file in text format. The allowed source file formats are zip and TextRecordInputStream.

The syntax is shown below:

hadoop fs -text <src>

hadoop fs -test

The hadoop test is used for file test operations. The syntax is shown below:
hadoop fs -test -[ezd] URI

Here "e" for checking the existence of a file, "z" for checking the file is zero length or not, "d" for checking the path is a directory or no. On success, the test command returns 1 else 0.

hadoop fs -tail

Hadoop tail command prints the last kilobytes of the file. The -f option can be used same as in unix.
> hafoop fs -tail /user/DPhadoop/sales.dat
12345 abc
2456 xyz

hadoop fs -stat

Hadoop stat returns the stats information on a path. The syntax of stat is shown below:
hadoop fs -stat URI
> hadoop fs -stat /user/DPhadoop/
2017-01-24 07:53:04

hadoop fs -setrep

Hadoop setrep is used to change the replication factor of a file. Use the -R option for recursively changing the replication factor.

hadoop fs -setrep -w 4 -R /user/DPhadoop/dir

hadoop fs -rmr

Recursively deletes the files and sub directories. The usage of rmr is shown below:
hadoop fs -rmr /user/hadoop/dir

hadoop fs -rm

Removes the specified list of files and empty directories. An example is shown below:

hadoop fs -rm /user/DPhadoop/file

hadoop fs -expunge

Used to empty the trash. The usage of expunge is shown below:

hadoop fs -expunge

hadoop fs -dus

The hadoop dus command prints the summary of file lengths
> hadoop fs -dus hdfs://namenodehost/user/DPhadoop
hdfs://namenodehost/user/DPhadoop 21792568333

hadoop fs -du

The du command displays aggregate length of files contained in the directory or the length of a file in case its just a file.

The syntax and usage is shown below:

hadoop fs -du hdfs://namenodehost/user/DPhadoop

hadoop fs -mv

It moves the files from source hdfs to destination hdfs. Hadoop mv command can also be used to move multiple source files into the target directory. In this case the target should be a directory.

The syntax is shown below:

hadoop fs -mv /user/DPhadoop/SrcFile /user/DPhadoop/TgtFile

hadoop fs -mv /user/DPhadoop/file1 /user/DPhadoop/file2 hdfs://namenodehost/user/DPhadoop/TgtDirectory

hadoop fs -moveFromLocal

The hadoop moveFromLocal command moves a file from local file system to the hdfs directory.
It removes the original source file. The usage example is shown below:
> hadoop fs -moveFromLocal products /user/DPhadoop/hadoopdemo

hadoop fs -getmerge

hadoop getmerge command concatenates the files in the source directory into the destination file. The syntax of the getmerge shell command is shown below:
hadoop fs -getmerge <src> <localdst> [addnl]

The addnl option is for adding new line character at the end of each file.

hadoop fs -get

Hadoop get command copies the files from hdfs to the local file system. The syntax of the get command is shown below:
hadoop fs -get /user/DPhadoop/hadoopdemo/hdfsFileName localFileName

hadoop fs -put

Hadoop put command is used to copy multiple sources to the destination system. The put command can also read the input from the stdin. The different syntaxes for the put command are shown below:

Syntax1: copy single file to hdfs

hadoop fs -put localfile /user/DPhadoop/hadoopdemo

Syntax2: copy multiple files to hdfs

hadoop fs -put localfile1 localfile2 /user/DPhadoop/hadoopdemo

Syntax3: Read input file name from stdin

hadoop fs -put - hdfs://namenodehost/user/DPhadoop/hadoopdemo

hadoop fs -cp

The hadoop cp command is for copying the source into the target. The cp command can also be used to copy multiple files into the target. In this case the target should be a directory.
The syntax is shown below:

hadoop fs -cp /user/DPhadoop/SrcFile /user/hadoop/TgtFile

hadoop fs -cp /user/DPhadoop/file1 /user/DPhadoop/file2 hdfs://namenodehost/user/hadoop/TgtDirectory

hadoop fs -copyToLocal

The hadoop copyToLocal command is used to copy a file from the hdfs to the local file system.
The syntax and usage example is shown below:
Syntax
hadoop fs -copyToLocal [-ignorecrc] [-crc] URI <localdst>

Example:

hadoop fs -copyToLocal /user/DPhadoop/hadoopdemo/sales salesdemo

The -ignorecrc option is used to copy the files that fail the crc check. The -crc option is for copying the files along with their CRC.

hadoop fs -copyFromLocal

The hadoop copyFromLocal command is used to copy a file from the local file system to the hadoop hdfs. The syntax and usage example are shown below:
Syntax:
hadoop fs -copyFromLocal <localsrc> URI
Example:

Check the data in local file
> ls sales
2000,iphone
2001, htc

Now copy this file to hdfs

> hadoop fs -copyFromLocal sales /user/DPhadoop/hadoopdemo

View the contents of the hdfs file.

> hadoop fs -cat /user/DPhadoop/hadoopdemo/sales
2000,iphone
2001, htc