Reverse Video in iOS

When I started writing the video processing engine for what would become the BeatCam app, one of the aspects we wanted to control was the direction in which the video traveled, e.g. forward or reverse. Manipulating media in iOS involves the use of Apple’s AVFoundation library and its related classes that allow developers to read, write, and process video and audio data for a variety of use cases, so that became the obvious place to start.

The premise was to able to script these behaviors dynamically and make them available for users to apply to their videos to create experiences like this:

For our needs, it wasn’t going to be possible to extract video frames in real-time in the order needed to convey reverse movement. The only possible solution involved the creation of a companion video file that contained a reverse ordering of the video frames of the original. Also, since BeatCam processes input audio sources using a completely different pipeline from the video, we could conveniently ignore any audio in the original and simply output a silent, reversed version of the original.

A nice solution to this problem is outlined by Andy Hin in which an AVAssetReader can be used to read the input frames and their corresponding timestamps. One can keep an array of the frame buffers, and when it’s time to write them to file, you use the same original timestamps in order, but use frame data starting at the end of the frame array and append to the output in time descending order.

However, Andy’s actual implementation is geared more toward reversing short videos of 30s or less. Longer videos will cause memory usage to blow up since one can’t keep a large number of frames around in memory before starting to output to file. And in BeatCam, any video in your library or that you shoot with the camera is fair game, so any implementation would need to function independently of input video duration.

In addition, this processing within BeatCam needed to occur behind the scenes since the user could be occupied with other aspects of the UI, so it was imperative that it run in its own thread. And if this call was now asynchronous, then some kind of completion block or delegate callback would need to be implemented.

One possible solution to the memory issue is to perform an initial reconnaissance of the entire file with respect to timestamps and frames, to organize this data into an array of passes where each pass contains a maximum number of frames (in this case, 100), and to then process each pass by extracting just the frames in the given pass and writing to output.

The Recon Stage

// main array to hold presentation times
NSMutableArray *revSampleTimes = [[NSMutableArray alloc] init];

// now go through the reader output to get some recon on frame presentation times
CMSampleBufferRef sample;
int localCount = 0;
while((sample = [assetReaderOutput copyNextSampleBuffer])) {
  CMTime presentationTime = CMSampleBufferGetPresentationTimeStamp(sample);
  NSValue *presentationValue = [NSValue valueWithBytes:&presentationTime objCType:@encode(CMTime)];
  [revSampleTimes addObject:presentationValue];
  CFRelease(sample);
  
  localCount++;
}

This method works only because AVAssetReader has the ability to read from input file for a given time range with little cost in initial seek time to the beginning of the range. Thus we can use the array of input frame timestamps to create the main array of dictionaries that will define the passes.

Creating the Array of Pass Info

// each pass is defined by a time range which we can specify each time we re-init the asset reader
    
// array that holds the pass info
NSMutableArray *passDicts = [[NSMutableArray alloc] init];

NSValue *initEventValue = [revSampleTimes objectAtIndex:0];
CMTime initEventTime = [initEventValue CMTimeValue];

CMTime passStartTime = [initEventValue CMTimeValue];
CMTime passEndTime = [initEventValue CMTimeValue];

int timeStartIndex = -1;
int timeEndIndex = -1;
int frameStartIndex = -1;
int frameEndIndex = -1;

NSValue *timeEventValue, *frameEventValue;
NSValue *passStartValue, *passEndValue;
CMTime timeEventTime, frameEventTime;

int totalPasses = (int)ceil((float)revSampleTimes.count / (float)numSamplesInPass);

BOOL initNewPass = NO;
for (NSInteger i=0; i0) {
      passStartValue = [NSValue valueWithBytes:&passStartTime objCType:@encode(CMTime)];
      passEndValue = [NSValue valueWithBytes:&passEndTime objCType:@encode(CMTime)];
      NSDictionary *dict = @{
        @"passStartTime": passStartValue,
        @"passEndTime": passEndValue,
        @"timeStartIndex" : [NSNumber numberWithLong:timeStartIndex],
        @"timeEndIndex": [NSNumber numberWithLong:timeEndIndex],
        @"frameStartIndex" : [NSNumber numberWithLong:frameStartIndex],
        @"frameEndIndex": [NSNumber numberWithLong:frameEndIndex]
      };
      [passDicts addObject:dict];
    }
    initNewPass = YES;
  }
  
  // if new pass then init the main vars
  if (initNewPass) {
    passStartTime = timeEventTime;
    timeStartIndex = (int)i;
    frameStartIndex = (int)(revSampleTimes.count - 1 - i);
    initNewPass = NO;
  }
}

// handle last pass
if ((passDicts.count < totalPasses) || revSampleTimes.count%numSamplesInPass != 0) {
  passStartValue = [NSValue valueWithBytes:&passStartTime objCType:@encode(CMTime)];
  passEndValue = [NSValue valueWithBytes:&passEndTime objCType:@encode(CMTime)];
  NSDictionary *dict = @{
    @"passStartTime": passStartValue,
    @"passEndTime": passEndValue,
    @"timeStartIndex" : [NSNumber numberWithLong:timeStartIndex],
    @"timeEndIndex": [NSNumber numberWithLong:timeEndIndex],
    @"frameStartIndex" : [NSNumber numberWithLong:frameStartIndex],
    @"frameEndIndex": [NSNumber numberWithLong:frameEndIndex]
  };
  [passDicts addObject:dict];
}

For each pass, a new AVAssetReader is initialized and the time range to be read is set to the subset of frames relating to the current pass.

Passes are read in reverse order, e.g. last pass to first, and in each pass, frames are written from the end of the pass to the start, which effectively writes all frames in the file in reverse order with capped memory usage (the number of frames in a pass) regardless of the length of the input video. This means the last pass will contain the total number of frames modulo the number of frames per pass. For example, if the input video has 456 total frames, and we are processing 100 frames per pass, then we will have 5 total passes, with the last pass containing 56 frames. This last pass will be the first one written to output, followed by 4 more of 100 frames a piece.

Using the Pass Array to Write the Output

int frameCount = 0; // master frame counter
int fpsInt = (int)(fps + 0.5);

// now go through the read passes and write to output
for (NSInteger z=passDicts.count-1; z>=0; z--) {
  NSDictionary *dict = [passDicts objectAtIndex:z];
  
  passStartValue = dict[@"passStartTime"];
  passStartTime = [passStartValue CMTimeValue];
  
  passEndValue = dict[@"passEndTime"];
  passEndTime = [passEndValue CMTimeValue];
  
  CMTime passDuration = CMTimeSubtract(passEndTime, passStartTime);
  
  int timeStartIx = (int)[dict[@"timeStartIndex"] longValue];
  int timeEndIx = (int)[dict[@"timeEndIndex"] longValue];
  
  int frameStartIx = (int)[dict[@"frameStartIndex"] longValue];
  int frameEndIx = (int)[dict[@"frameEndIndex"] longValue];
  
  CMTimeRange localRange = CMTimeRangeMake(passStartTime,passDuration);
  NSValue *localRangeValue = [NSValue valueWithBytes:&localRange objCType:@encode(CMTimeRange)];
  NSMutableArray *localRanges = [[NSMutableArray alloc] init];
  [localRanges addObject:localRangeValue];
  
  // reset the reader to the range of the pass
  [assetReaderOutput resetForReadingTimeRanges:localRanges];
  
  // read in the samples of the pass
  NSMutableArray *samples = [[NSMutableArray alloc] init];
  while((sample = [assetReaderOutput copyNextSampleBuffer])) {
    [samples addObject:(__bridge id)sample];
    CFRelease(sample);
  }
  
  // append samples to output using the recorded frame times
  for (NSInteger i=0; i= revSampleTimes.count) {
      NSLog(@"%s pass %ld: more samples than recorded frames! %d >= %lu ", __FUNCTION__, (long)z, frameCount, (unsigned long)revSampleTimes.count);
      break;
    }
    
    // get the orig presentation time (from start to end)
    NSValue *eventValue = [revSampleTimes objectAtIndex:frameCount];
    CMTime eventTime = [eventValue CMTimeValue];
    
    // take the image/pixel buffer from tail end of the array
    CVPixelBufferRef imageBufferRef = CMSampleBufferGetImageBuffer((__bridge CMSampleBufferRef)samples[samples.count - i - 1]);
    
    // append frames to output
    BOOL append_ok = NO;
    int j = 0;
    while (!append_ok && j < fpsInt) {
      
      if (adaptor.assetWriterInput.readyForMoreMediaData) {
        append_ok = [adaptor appendPixelBuffer:imageBufferRef withPresentationTime:eventTime];
        if (!append_ok)
          NSLog(@"%s Problem appending frame at time: %lld", __FUNCTION__, eventTime.value);
      }
      else {
        // adaptor not ready
        [NSThread sleepForTimeInterval:0.05];
      }
      
      j++;
    }
    
    if (!append_ok)
      NSLog(@"%s error appending frame %d; times %d", __FUNCTION__, frameCount, j);
    
    frameCount++;
  }
  
  // release the samples array for this pass
  samples = nil;
}

You can see this in action in the app’s video filters as exemplified by some of the posts on BeatCam’s Instagram page

The class used for creating reverse video, along with a sample XCode project is available on Github. I hope you find it useful.

One aside: if you do find that you need to include a reversed version of the audio, then you will need to extract the audio from the video, reverse the order of the audio samples (make sure you take mono vs stereo into account), and then export a new video using the newly created reversed video and audio tracks.

The export, in this case, is achieved by creating a new AVMutableComposition, defining the tracks, length, and other details, and then exporting to file using AVAssetExportSession. Consult Apple’s documentation for a full explanation of how this is achieved.

NFL Football and Raisins

It’s no secret that at this time of year, my (un)healthy addiction to NFL football rears its ugly head. I’m a lifelong Giants fan, which means that by law, I have to root against all other NFC East teams, and later in the season, against any other NFC teams that might prevent said G-Men from landing a playoff spot.

My daughter Alana is 5, and is just starting to discover professional sports and the differences between them. So last week, I thought she might have an interest in a 4pm game between the Eagles and the Ravens. The dialogue went something like this:

Me: Alana, who do you want to win – the Eagles or the Ravens?
Alana: The Eagles or who? The Raisins …?
Me: No, the Ravens
Alana: (thinking) … I want the Raisins to win
Me: No, no, it’s the Ravens, not the Raisins
Alana: (spots closeup of Haloti Ngota) Wow! Look at that giant Raisin!

I’m now hoping that the Ravens continue to be the Raisins for a few years to come, as it is now standard practice to find out who the Raisins are playing this week. Turns out it was the Patriots in a hard-fought contest last night, and now the Browns this coming Thurs.

Knicks’ 2011-12 schedule – Knicks Blog

Says Ian Begley:

“Knicks vs. Heat, April 15: LeBron James says he doesn’t want to play the “villain” role anymore, but he won’t be able to avoid it on this night. James will be making his third trip to the Garden since spurning the Knicks [and others] to form a big three with Dwyane Wade and Chris Bosh. This matchup is Game No. 60 of the season, so it could have serious playoff implications.”

via Rapid Reaction: Knicks’ 2011-12 schedule – Knicks Blog – ESPN New York.

This will be a good one to revisit in April …

Eclipse and SFTP

One of my students at ITP just asked about file sync between one’s local file system to a remote file system using Eclipse and SFTP, so I put together a quick how-to, which might be of interest. The actual text/steps may vary, depending upon your version of Eclipse, but all the basics are there.

The one caveat of this process is that I’ve only been successful in setup and sync with an existing remote directory, and then importing it locally, not the other way around, e.g. exporting local to remote (though I have not tried very hard in this area — for whatever reason, the target has always existed whenever I’ve set up sync). So a remote target folder of some kind must already exist before you sync it to a local one, if you’re using these instructions.

To equate a local directory with a pre-existing remote one:

  1. Help > Software Updates
  2. Choose Available Software Tab
  3. Click Add Siteā€¦, and then put http://eclipse.jcraft.com/ for the URL field.
  4. Choose SFTP Plug-in, and then click Install …, and then restart after the install
  5. Using Project Explorer view, make a new local project
  6. Create a folder within that project that will map to a remote folder
  7. Right click on the local folder in Project Explorer and choose Import
  8. In the Select dialog, choose Other > SFTP
  9. Choose the local folder as the one to receive the import
  10. Create a new SFTP site and give your host and login info
  11. If successfully connected to remote site, you should get the remote directory tree
  12. Choose the remote folder to import
  13. The remote files should now be in the local folder

Now to keep sync between these two:

  1. Go to Window > Show View > Other… > Team > Synchronize
  2. Click the first upper right icon in the Synchronize tab
  3. Choose SFTP from the Synchronize dialog
  4. The previously mapped folder should appear with all files selected
  5. Click Finish
  6. Eclipse will ask you if you want to use Team Synch perspective. Personally I don’t use this. I just keep the Synchronize view as part of my workbench

To test:

  1. Make a change in a file within the local folder
  2. It should then appear in the Synchronize view in Outgoing mode
  3. Using the buttons at the top of the Synchronize tab, or the right-click menus on any of the items that have changed, you can upload or check the diff between the local and remote counterpart

So there you have it. For projects that involve multiple people, I would recommend using a version control system like Subversion, but for quick one-person, one code-base jobs, this works nicely …