module Outatime

Outatime::Fetcher is responsible for finding the exact revision for each file for a given time.

AWS S3 API lists all file revisions in a very particular order and this class takes advantage of that to quickly parse and find the revision.

The 'GET Bucket Object Versions' (docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETVersion.html) returns all file revisions ordered by key (path + filename) and last modified time.

For example:

id: 12456, key: “src/file1”, last_modified: 7 Feb 11:38 id: 12357, key: “src/file1”, last_modified: 7 Feb 11:37 id: 12357, key: “src/file1”, last_modified: 7 Feb 11:00 id: 22222, key: “src/file2”, last_modified: 7 Feb 11:39 …

Keep in mind that this response is paginated, where the max amount of revisions per page is 1000.

When a versioned bucket contains a huge amount of files and revisions, fetching all data may take a long time. So Outatime::Fetcher starts downloading the correct revision even when not all file revisions are already fetched, i.e. when it is still downloading the revisions and their data. That accelerates the process.

So how do we know when we have all information we need before start downloading a specific file revision? As the response is ordered by key (filename) and timestamps, fetcher can start downloading if the actual response page contains all possible revisions for a file, i.e. its revisions aren't paginated.

When the response for a given key (filename) is paginated, the file is not downloaded until all revisions are fetched in the next response page.

This algorithm removes the need to fetch all file revisions (which may take several requests) for a versioned bucket before starting to download its files, but acts on each individual file instead.