File Handling in Bash Scripting

When writing bash scripts, it's common to need to work with files and directories. This includes tasks like reading and writing file contents, checking if a file exists, renaming or deleting files, and getting file metadata like permissions and size.

This article will demonstrate bash scripting techniques for effectively handling files and directories with well-commented examples.

Reading Files

A common task is reading the contents of a file into a variable in your script. This allows you to store the file contents and process it later.

The cat command prints a file's contents to standard output. We can use command substitution to assign that output to a variable:

#!/bin/bash

# Store contents of the file in a variable
file_contents=$(cat file.txt)

echo "File contents: $file_contents"

This reads file.txt and assigns it to $file_contents. We echo the variable to print the contents.

For larger files, cat may be inefficient. An alternate is reading a file line-by-line with a while loop:

#!/bin/bash

# Read file line-by-line
while read line; do
  echo "$line" 
done < "input.txt"

This loops through each line of input.txt, assigning it to $line and echoing it.

Writing to Files

To write to a file, use the > redirect operator:

#!/bin/bash 

# Write output to file
echo "This is some text" > outfile.txt

This writes "This is some text" into outfile.txt, overwriting any existing content.

To append instead of overwrite, use >>:

#!/bin/bash

# Append output to the end of file 
echo "Appending this line" >> appendfile.txt

You can also use tee to write to a file and also print to standard out at the same time:

echo "This gets printed and logged" | tee logfile.txt

Checking if a File Exists

To avoid errors, check if a file exists before accessing it:

#!/bin/bash

# Check if file exists
if [ -f "file.txt" ]; then
  echo "File found" 
else
  echo "File not found"
fi

The -f flag for the if condition checks if the file exists and is a regular file. Other test flags include:

-d - Directory exists
-e - File exists (regular file or directory)
-s - File exists and size is greater than 0

Getting File Metadata

You can get useful metadata about files like size, permissions, etc.

For example, get a file's size in bytes:

#!/bin/bash

# Get file size
filesize=$(stat -c%s "file.txt")
echo "Size: $filesize bytes"

The stat command returns file info, and %s prints just the size.

To get a file's permissions:

#!/bin/bash

# Get file permissions
perms=$(stat -c%A "file.txt")
echo "Permissions: $perms"

%A prints the permissions in symbolic format. Other useful formats are %U for the user and %G for the group.

Renaming and Deleting Files

You can rename or delete files and directories from within scripts.

To rename a file:

#!/bin/bash

# Rename file 
mv oldname.txt newname.txt

To delete:

#!/bin/bash

# Delete file
rm filename.txt

And to remove a directory and all its contents recursively:

#!/bin/bash

# Delete directory recursively
rm -r directoryname

Other File Handling Commands

There are a few more useful bash commands that can be included when covering file handling in bash scripting:

cp - Copy files and directories. Useful for backing up files or setting up test data. Example:
```
  cp file.txt file_copy.txt
```
mkdir - Make new directories. Often needed to create output directories for scripts:
```
  mkdir /path/to/newdir
```
find - Search for files recursively matching given criteria. Helps locate files based on name, size, permissions, etc:
```
  find /home -name "*.txt"
```
head/tail - Read first or last N lines of a file. Useful for previewing file contents:
```
  head -n 10 file.txt 
  tail -n 5 file.txt
```
touch - Update file timestamps or create empty files. Can force a file update:
```
  touch file.txt
```
chmod - Change file permissions like make executable:
```
  chmod +x script.sh
```
gzip/gunzip - Compress and extract gzip files:
```
  gzip file.txt
  gunzip file.txt.gz
```
tar - Archive multiple files and dirs into a tar file:
```
  tar -cvf archive.tar /path/to/files
```

Reading CSV files:

  # Read CSV file line-by-line
  while IFS="," read -r col1 col2 col3; do
    echo "$col1 | $col2 | $col3"
  done < file.csv

Use IFS (input field separator) to split CSV rows into columns.

Parse XML/JSON with jq:
```
  # Extract JSON value
  name=$(cat file.json | jq -r '.name')
```
jq allows easy JSON parsing to extract values in scripts.

File globbing - Match patterns with * and ?:

  # Copy files matching pattern
  cp /path/*.txt /backup/

Check file contents with grep:

  # Grep for match in file
  if grep -q "some_text" file.txt; then
    echo "Found"
  fi

Use grep to search file contents.

Create temp files and dirs:
```
  # Create temp file
  tmpfile=$(mktemp)
```
mktemp makes disposable temp files.
Stdout/stderr redirection:
```
  # Redirect stdout and stderr 
  program >out.txt 2>err.txt
```
Send stdout and stderr output to different files.
Named pipes for inter-process communication:
```
  # Read from named pipe
  mkfifo mypipe
  cat mypipe
```
Multiple processes can communicate via named pipes.

These provide additional file and directory manipulation capabilities that can be useful in bash scripts. The key concepts remain using redirects, loops, command substitution, and conditional checks for robust file handling.

So in summary, bash provides many useful commands for reading, writing, checking files, getting metadata, renaming/deleting and more. Mastering file handling will enable you to write more powerful scripts.

The key is knowing which commands like cat, redirects, stat, mv, rm to use for different file tasks. Refer to this article for syntax examples.