Secure File Transfer Protocols

There are two widely used secure file transfer protocols that are confused quite often. The talk is about SFTP and FTPS. Although the purposes (and the names ;)) of the protocols are quite similar, they are rather different and, what is important, incompatible with each other. I will concern the main differences between the protocols in this article.

SFTP (SSH file transfer protocol) runs as a subsystem on top of secure SSH channel. All the traffic is always encrypted, the security is guaranteed by the underlying SSH protocol.

FTPS (FTP over SSL/TLS) is an improved version of classic FTP protocol, where TCP connection is optionally secured with SSL. In all other aspects the protocol is compatible with good old FTP.

What is interesting is that neither "SFTP" nor "FTPS" is an official name of the corresponding protocol. There’s even no “complete” standard for SFTP – all the versions of this protocol are only available as internet-drafts. FTP over SSL is particularly defined in RFC 2228, however, most of existing implementations offer much more features than are defined in that RFC (for instance, “implicit FTP” term was invented by the developers of one of the FTPS products, such a term has been never officially defined). Thus, both protocols are actually standards de facto, but not de jure. This fact causes a lot of (in)compatibility problems between different software implementations.

Let’s shortly consider the differences and the features common for both protocols.

1. Transport

SFTP runs over secure SSH channel. Moreover, one can establish several parallel SFTP connections through a single SSH (and TCP) connection. Moreover, one can additionally run a remote shell through the same SSH channel. Security of FTPS is optionally guaranteed by SSL/TLS protocol. Each FTPS connection requires a separate TCP connection to be established to the same server. Besides, transfer of a file requires a separate TCP connection, called data connection. FTPS works in either active (client opens a listening data port and server connects to it) or passive (vice versa) mode.

2. Security features.

Obviously, security features of the both file transfer protocols rely on security features of the underlying security protocols. Both SSH and SSL provide the following security features: server authentication (mandatory in both protocols), client authentication (optional in SSL, mandatory in SSH), strong key exchange, data encryption and integrity protection. Strengths of the algorithms used in both protocols are approximately the same. However, unlike SSL, which allows only certificate-based peer authentication, SSH provides support for a number of possible client authentication methods, such as password-based, keyboard-interactive and public key-based.

3. File transfer features.

In general, SFTP is more flexible than FTPS. In particular, SFTP provides access to remote files as if they were local (via Open/Read/Write/Close functions), emulating free access to them. Besides, SFTP declares a standardized way for file attributes requests. It is known that FTP does not provide a unified way to request a file list; the developers of FTP clients are forced to implement reading of dozens of different file list formats to make their implementations compatible with as many existing servers as possible. Actually, there’s a reason for it – FTP originally has been designed as human-readable protocol, not machine-readable.

4. Firewall friendliness.

FTP is known to be not a firewall-friendly protocol. This is caused by the necessity of establishing a second connection for data transfer. This fact resulted in invention of different secure/insecure mode switches (such as PROT and CCC commands) and other crutches. As SFTP does not require secondary connections, it does not have problems with proxies and firewalls.

5. Extensibility.

Both protocols can be extended with new commands.

As you see, there are much more reasons for using SFTP rather than FTPS. There are only two features that may put SFTP behind FTPS:
* SSL provides standardized support for X.509 certificate authentication. SSH does not support X.509 certificates natively, making PKI infrastructure hardly used with SFTP.
* FTPS provides independent encryption modes for control and data channels, what might be suitable for logging and monitoring purposes.

If you are interested in more details about the differences between SFTP and FTPS, please see the following article.

Besides FTPS and SFTP, there do exist other, more complex, transfer protocols, such as OFTP. However, due to low popularity and specifics of use of such protocols, comparing them to mainstream FTPS and SFTP protocols is not correct enough. I will concern OFTP and its main features in one of the future posts.

Another stimulus to write secure applications (taken from the Washington Post):

A data breach last year at Princeton, N.J., payment processor Heartland Payment Systems may have compromised tens of millions of credit and debit card transactions, the company said today.

If accurate, such figures may make the Heartland incident one of the largest data breaches ever reported.

It is not clear at the moment what exactly has caused the breach. The paper talks about some malicious piece of software, however, it says nothing about how had this code got to the payment processing network. In any case, a lack of attention or qualification (hope, the former) of developers and security officers has resulted in serious damage for the Heartland Payment Systems' reputation.

The data stolen includes the digital information encoded onto the magnetic stripe built into the backs of credit and debit cards. Armed with this data, thieves can fashion counterfeit credit cards by imprinting the same stolen information onto fabricated cards.

On the significance of authentication

As I said in the very first post, many developers think that it is very easy to make their software applications secure. One should take a software component that implements the necessary security protocol, plug it to the application and have the application robust, reliable and secure. Though such straightforward approach may work for simple applications, it is very dangerous to use it for securing complex systems. Why dangerous? Because it only gives the illusion of security. You think that your system is secure, while it actually isn’t.

Let’s consider TLS. What can you say about it? TLS is a transport-layer protocol that secures TCP data by encrypting it with robust industry-standard algorithms. A malicious person who intercepts network traffic sees just a sequence of meaningless characters and not the original message.

"Encryption" word sounds like a spell for many people. "If my data is encrypted, no one can read it." Thinking this way is a fatal flaw. No one can read it if the key is in safe place.

Besides encryption, TLS offers a pack of other security services that are often ignored. One of them is server authentication.

Basically, to authenticate someone is to ensure that this someone is the person he pretends to be. When I was ten, I authenticated my friend Michael by three consecutive knocks on the door. The bank guard authenticates the officer by comparing his face to the photo on his PhotoID. Your computer authenticates you by prompting you to provide your account name and password.

In TLS, authentication is performed with the use of digital certificates. Each TLS-capable server has an associated certificate which contains information about its owner, such as host name (or IP address), organization name and its geographic location. Certificates are issued by a trusted insitutions called certification authorities (CA). Three widely used certification authorities are Thawte, Verisign and Comodo.

CA is not required to be a worldwide organization with thousands of employees. To be exact, any person who has a computer can become a CA and issue certificates. However, who will trust them? If my friend Michael told about three secret knocks to everyone, would I consider the knocker to be trusted?

How the authentication is performed? Besides the certificate, the server possesses a corresponding entity called a private key (which can be imagined like a very long binary sequence). Certificate and private key are complementary entities that can be used to perform complementary cryptographic operations. Private key can be used for signing data, while the corresponding certificate can be used for validating the correctness of such signatures. Certificate can be used for data encryption, while only the corresponding private key is able to decrypt data encrypted with such a certificate.

It is obvious that unlike the certificate that is publicly available, the private key must be kept in secret by its possessor.

To authenticate himself to the client, the server uses his private key to sign an arbitrary chunk of data provided by the client. The client then uses server’s certificate to validate the correctness of the signature. As nobody else knows is supposed to know the private key of the server, correctness of the signature ensures that you are talking either to a real server or to someone who has stolen his private key. I do not consider the latter here, this case will be discussed in future posts.

TLS also provides support for client authentication. It works exactly in the same way with the only difference that the client signs the data and the server verifies the correctness of his signature using client’s certificate.

The problem with TLS is that a lot of people who use it think that encryption is its primary feature, while authentication is a secondary one. It is absolutely not correct! There’s no sense in locking the safe if the guard opens it for everyone who asks him about it. If your application does not care about server authentication, it is vulnerable to a dozen of different attacks. One of the most common and easy-to-use attacks based on the lack of authentication is server substitution. Someone just creates a fake server in the network that identifies himself as the original server (there’s a million ways to do this, I omit the exact how-to. Just keep in mind that it is much easier than it might appear at first glance). As no authentication is performed, you will never understand that you are talking to the wrong server. What is worse, the illusion of security is preserved. You are using TLS, so you are secure, aren’t you?

The security of the overall system is defined by the security of its weakest component. There’s no sense in locking the door if the window is open.

Why authentication is often ignored by the developers? I think there are three general reasons for this.

First, authentication is not something one can touch. Encryption is an obvious thing. Authentication is not as transparent as encryption. Besides, 8 persons of 10 consider that “security” is equal to “encryption”.

Second, implementing correct authentication procedure is not an easy task. Some developers (and PM’s/CTO’s as well) consider this task too expensive and do not want to spend a lot of time in implementing it correctly. “Everything works fine without it, so why do we have to postpone the release date of our product for two months for the sake of the feature that no one will have a chance to see?”. Was it a long time ago you’ve heard about 10000 credit card number hijacking the last time? It is likely that their system “worked fine” too.

The third one, the lightmost reason, is the [absolutely reasonable] wish of developer to make the software as much user-friendly as possible. Authentication often requires interaction to the user. Sometimes it is necessary to ask user if he trusts the particular web site or application, sometimes another warning should be displayed. Some developers think that this will bother user. Will not the user be bothered by the stolen password database?

Authentication of the remote is absolutely necessary. The most impenetrable door will provide no help if everyone has a key to unlock it. The most progressive encryption algorithm is worth nothing if you are exchanging encrypted data with a wrong person.

Offtopic: Merry Christmas and Happy New Year. Have a nice '09!

A very good article has been published by CWE. Top 25 Most Dangerous Programming Errors must be read by any software developer who seeks to develop secure and, what is more, safe applications.

The article divides all the programming errors in three classes, (a) insecure interaction between components, (b) risky resource management and (c) porous defences. Top 25 includes both good old errors (such as buffer overrun), web-specific XSS and SQL injection errors, and issues caused by the lack of attention to application's security.