(or “Why File Synchronisation is Hard”)

One of the questions I get asked most often is why it’s not currently possible to move files or folders in UX Write. While this might seem like a very simple feature to add, there’s actually a lot more to the story, due to the need to ensure reliable file synchronisation in which no data loss can ever occur.

Normally, this is all handled by the operating system and third-party sync clients like that provided by Dropbox. However, Apple made the tremendously unfortunate decision to exclude any notion of a user-visible filesystem on iOS, so this isn’t possible. Desktop operating systems have had built-in file management capabilities for more than thirty years, and everyone who’s ever used a personal computer is familiar with the concept of organising their files into different folders.

While unnecessary for casual usage like browsing the web or playing games, file management is absolutely critical for content creation and management; its omission from iOS cripples the iPad as a productivity device. Nonetheless, it is possible for individual app developers to work around this limitation, but doing so is highly non-trivial in the case where synchronisation with an online file storage service is needed.

For the developer, moving files locally on a device is trivial to implement. There’s programming interfaces built in to the operating system to do this, and it’s a simple task to create a user interface for selecting the files to move and their destination. The difficulty lies in synchronising those changes with a server (or “The Cloud”, if you prefer that term) in such a way that guarantees no data gets lost in the process.

With online storage services, it’s possible to have multiple devices and people with access to a particular account. Software like the Dropbox desktop client, and UX Write itself, allow changes to be made to files while you are offline, and then later push those changes to the server. However, cases can sometimes arise in which different people have changed or moved the same file or folder while offline, and then later reconnect to the Internet and sync with the server. This is where things can go disastrously wrong, if conflicts are not handled carefully by the software.

An Example

Suppose Bob and Alice have access to a shared account, and are working with the same set of documents relating to a particular project. Both are working offline, when Bob decides to move a folder to a different location and Alice simultaneously makes some changes to a document inside that folder.

When they subsequently reconnect to the Internet, their iPad or PC will try to sync with the sever. Whoever happens to sync first will “win” and have their changes applied to the server successfully, while the “loser” will run into errors, with the server reporting that the folder doesn’t exist any more.

If Bob wins, the folder will be moved, but when Alice’s device tries to upload her modified version, it will encounter an error because the folder it was in no longer exists at its previous location. An incorrectly-implemented syncing mechanism may conclude that because it can no longer see a folder at that location on the server, it must have been deleted and that it should therefore result in the local copy being deleted as well — which happens to contain Alice’s modified document. The syncing mechanism would then see what appears to be a completely new folder on the server, and download that. But Alice’s changes would be gone.

Solutions

File synchronisation is a complex topic, but necessarily has to be addressed to at least some extent by any app that lets you work offline with documents whose primary copy resides online. It’s unacceptable for an app to blindly push changes to a server without checking to see if the files or folders in question have been modified by someone else, and it’s also unacceptable for the app to download all changes from the server and overwrite local copies that may have been modified independently. It’s necessary to provide both conflict detection and resolution capabilities, with the latter preferably involving automatic merging of changes where possible.

There are many possible solutions to these problems, which have been implemented in various syncing clients and version control systems. Git provides what I consider to be the most elegant solution to this (plus a whole host of other things), but that’s another topic entirely which I won’t get into right now.

The simplest approach I’ve been considering for an initial implementation of moving is to do it synchronously — that is, require you to be online at the time of the move, and wait until the server has reported a successful completion of the operation before allowing you to continue using the program. In computer science terms, this is known as an “atomic test and set” operation, where a check is performed to see if the change is valid, and if so, the change is made, all in one indivisible operation, avoiding race conditions. The alternative is to perform the move asynchronously — that is, in the background at a later point in time; but that has all the problems mentioned above, and requires keeping track of lots of extra information in order to do it reliably.

In the long term, I plan to integrate support for Git directly into UX Write, so you’ll have access to its full set of functionality, including versioning, branching & merging, and conflict resolution. Software developers use this all the time for collaboration, and I’d encourage you to learn more about it if you’re interested. I’ll be posting more about this integration in the future once I’ve had a chance to think it through further.

Doesn’t iCloud handle all of this?

Sort of, but only in the context of a single application and a single user. Since UX Write only runs on the iPhone and iPad, many people use it in conjunction with other applications like Microsoft Word on their desktop, so it’s necessary to sync the files between different platforms. iCloud has a restriction that only allows files to be accessed from within the application that created them, and provides neither a generic filesystem model of the sort found on Mac, Windows, or Linux, nor does it cater for the need to edit a given document with different applications. So that basically rules it out, as far as I’m concerned.

In Summary

Better file management facilities are coming to UX Write, but as described above these aren’t straightforward to provide in the context of the ecosystem we have today where everything is stored and synced online across multiple devices. At the moment, my primary focus is on building a word processor, not a file manager, so the latter has secondary priority. But I am very much aware of the need for improvements in this area, and will get to them eventually.