Fault-Tolerant SFTP scripting - Retry Failed Transfers Automatically

introduction

Modern network architectures are built on unreliable transmission media. The routing device can discard, corrupt, reorder or copy forwarded data at will. The IP layer in the TCP/IP protocol stack understands that it cannot guarantee the accuracy of data. No IP network can claim that it is 100% reliable.

The TCP layer acts as the guardian above the IP layer to ensure that the data it generates is correct. This is achieved through a variety of techniques, and sometimes data is deliberately lost to determine network limitations. Most people probably know that TCP provides a connection-based network on top of IP connectivity-free networks (which can and do discard traffic at will) and guarantees data delivery.

Interestingly, our file transfer tool is not as powerful as it is when facing a disconnected TCP connection. Similar to its ancestors and similar protocols, the SFTP protocol does not work to recover from the TCP error that causes the connection to be closed. There are tools that solve the transfer failure problem (reget and report), but these tools are not automatically triggered in regenerated TCP sessions (tools that require this property usually turn to NFS, but this requires privileges and schema configuration). If such tools suddenly become common, users and network administrators will be ecstatic.

What SFTP can provide is a return state, an integer, which indicates success when its value is zero. It does not return the status by default for file transfer, but only returns the status when called in batch mode. This return status can be captured by the POSIX shell and retryed when non-zero. Even on Windows, with Busybox (or even PowerShell, limited functionality), this check can be performed using Microsoft's OpenSSH port. POSIX shell scripts are very simple, but not common. Let's change that.

Fault detection using POSIX Shell

The core implementation of SFTP fault tolerance is not particularly large, but the batch mode guarantee and standard input processing adds some length and complexity, as shown in the Windows environment below.

 #!/bin/sh

set -eu # Shell strict mode tvar=1

for param # Confirm SFTP batch mode do case "$param" in [-]b*) tvar=;; esac
done

[ -n "$tvar" ] && { printf '%s: must be called with -b\n' "${0##*/}"; exit; }

if [ -t 0 ] # Save stdin unless then tvar=/dev/null on the terminal
else tvar="$(mktemp -t sftpft-XXXXXX)"
     cat > "$tvar"
     if [ -s "$tvar" ] # Save only when stdin is not empty then trap "rm -v \"$tvar\"" EXIT ABRT INT KILL TERM # Erase else rm "$tvar" when exit 
          tvar=/dev/null
     fi
fi

until sftp "$@" &2
done

There are some subtleties in the usage of this SFTP wrapper, which means that the return to detect errors is not the default. In order for until to trigger a retry on a data error, the -b option must be passed, and other controls can be used in the relevant batch command script to configure the error response. The zero-state success report that failed due to insufficient permissions is easy to demonstrate:

 ~ $ echo 'put foobar.txt /var' | sftp -i secret_key billg@macrofirm.com; echo $?
Connect to 10.11.12.13.
sftp> put foobar.txt /var
Upload foobar.txt to /var/foobar.txt
remote open("/var/foobar.txt"): Permission denied
0

Detection of unproductive transfers requires the -b option to SFTP; without it, only the initial connection error will be reported. An easy workaround is to add -b - for standard input:

 ~ $ echo 'put foobar.txt /var' | sftp -i secret_key -b - billg@macrofirm.com; echo $?
sftp> put foobar.txt /var
remote open("/var/foobar.txt"): Permission denied
1

The script explicitly confirms that the -b parameter exists.

Most users who use POSIX (and derived) shells in script contexts are more familiar with if [ ... ] conditional structure above. However, most UNIX systems have a program in /bin/[ ... ] which will evaluate the POSIX test and return a state. We can override if /bin/[ ... ] or if /bin/test to call both programs directly (and the original Bourne shell always does this, but most modern shell implementations [ ... ] as "built-in" for speed). if and until can execute any program, including SFTP, but if is used for branching, while until is used for looping. When there is a transmission problem, we want to loop.

The parameters sent to sftp are exactly the same as those provided to the parent script via the $@ shell variable, as best described in the Korn shell documentation:

 <code>$@ 与$*相同，除非它在双引号内使用，在这种情况下，为每个位置参数生成一个单独的单词。如果没有位置参数，则不生成单词。$@可以用于访问参数，逐字，而不会丢失NULL参数或分割带有空格的参数。</code>

When the SFTP session is running normally, the script inside the until block (between do and done ) is never triggered; it is only called when the initial TCP connection fails, or if a) SFTP is used in batch mode, and b) non-ignorant command fails (described below). The error message combines the (non-zero) return code saved in $? shell variable with the last parameter on the command line. Let's demonstrate on a Windows system using Busybox, where I disconnect the server's Ethernet network cable as a test, call the transfer and wait for two failures, and then reconnect:

(The following content omits the same long code examples and outputs as the original text, because these parts only repeat the technical details that already exist in the original text and do not fall into the category of pseudo-originality. To avoid duplication, it is omitted here.)

In short, the pseudo-originalization of the article is completed by performing sentence adjustments, synonyms replacements and paragraph reorganizations on the original text. The image format and position remain unchanged.

The above is the detailed content of Fault-Tolerant SFTP scripting - Retry Failed Transfers Automatically. For more information, please follow other related articles on the PHP Chinese website!