What is a data wrangler?
A data wrangler, also often called a digital loader or a data manager, is one of the core camera department positions introduced in filmmaking with the advent of digital capture. Put shortly, the main task of data wranglers is to download the footage and metadata from the camera magazines to hard drive arrays or servers. While this may sound simple, a lot can go wrong, and the stakes are high. Remember, once the magazine is out of the camera, the work of the whole cast and crew lies in the hands of the data wrangler.
Often viewed by producers as an entry position, data wrangling is no place to cut corners. A misplaced click can erase days or weeks of work, not to mention the monetary damages. What’s more the data wrangler is often the last person on set to handle the image data. It’s usually not until the end of the day when the drives are shipped to the editorial that anyone will check the work of the data wrangler, and then it’s usually too late to do anything. This is why data wranglers have strict protocols and routines to ensure the footage and metadata make it to the editing room safe and sound. These include specialised software, multiple backups and teamwork involving the whole camera department.
So, after this brief overview of what it takes to be a data wrangler, let’s dive deeper.
What does a data wrangler actually do?
Like all film jobs, data wrangling duties and responsibilities can vary widely from production to production. A small one-camera commercial shoot or a guerrilla style documentary have quite different workflows than, say, the second unit of a major Hollywood action blockbuster sporting 15 cameras. What’s more, it’s often not the production itself, but the production company that dictates how the data wrangler goes about their job. Bigger companies like Netflix and Disney have very strict guidelines when it comes to data management, designed not only for the safety of the footage, but also for confidentiality and efficient collaboration between team members that are often on different continents. However, barring these extreme examples, let’s see what a typical data wrangler list of responsibilities looks like:
Getting footage off the mags
As the digital loader’s key task, naturally we start with this. Downloading footage from a memory card can seem deceptively simple: just copy and paste, right? I’m afraid it’s not so easy. To start, most professional cinema cameras’ memory drives use proprietary file systems and formatting, so they require specialised software before the computer can even recognise them. To make matters more complicated, a camera firmware update or a new software release can completely change the download process, so a data wrangler must always be on top of new developments in the industry. This is why it’s good practice to have the data wrangler on at least one of the prep days for any production.
For the actual download, even if it’s available, copy and paste is rarely the best option. Data wranglers prefer software like Silverstack, ShotPut Pro or Hedge. These tools not only copy the data in the most efficient way, but they also verify the data integrity of the downloaded material. What this means is that software checks that each copy is a bit-to-bit exact copy of the original. This is done to find and correct any read errors or hardware malfunction that can result in corrupted data. The software also supports simultaneous download to multiple drives, facilitating good backup practices and saving time.
This brings us to another aspect of the download process – speed. You may be surprised to hear this, but most camera packages do not come with that much storage. It’s typical to receive an Alexa Mini with just three or four 256GB cards, each good for just under 15 minutes of Open Gate 3.4K RAW footage. In a fast paced shoot these can fill up quite quickly, and when they do, the camera cannot roll anymore and production grinds to a halt. You can see how this can put the data wrangler under a great deal of pressure to keep up with the pace of the shoot, while not cutting corners in terms of data integrity.
There is no room for errors. Once the data wrangler gives the card back to the camera assistant, it will be wiped and reused. When a card has been overwritten no previous data can be restored from it. What’s more, in certain cameras like the Arri Alexa Mini even a simple quick format, without any overwriting, can render any information on the card unrecoverable.
Managing backups
The second key component of a digital loader’s list of responsibilities is managing the backups, or the multiple copies of the image data. Depending on the size and scope of the production, there can be many different backup protocols for the data manager to follow.
Smaller, one day shoots generally just rely on multiple copies made at the time of download. As discussed above, professional copy verification software that data wranglers use is able to simultaneously replicate and verify data on multiple drives. Generally, at the very minimum the footage must me copied and verifies on two drives before the card is released for reuse. However, few data managers will be happy with less than three, so that’s kind of an industry standard.
At the end of the day, all that’s left of a shooting day that may cost hundreds of thousands of dollars is the data on these few drives. Unsurprisingly, these are the most important bits of kit when the lights go out. The usual practice is for the drives to go separate ways at the end of a shoot, so that the chance of a car accident or a theft wiping out everyone’s hard work is minimised. Where each drive goes depends much on the production type and the particular production company’s workflow.
Generally, one drive, usually referred to as rushes is taken back to the editorial, where it is copied over to a workstation or a server for transcoding and further work. In remote locations, this can involve shipping the drive with a courier service overnight, although nowadays it’s mostly uploads over the internet. Once in editorial, the footage is multiplied yet again, so the data wrangler can breathe a little easier. It’s usually at this point that any secondary backups are done – many production companies, for example, rely on LTO tapes for their long-term archives.
Data wrangler vs DIT
These two positions often get mixed up, mostly because they both involve manipulating footage on computers, but also because they are sometimes sort-of combined on smaller jobs. However, a data wrangler and a DIT (digital imaging technician) are two separate and very distinct positions. As discussed above, data wranglers are mainly responsible for the physical handling of the footage – the transfer of image data from the camera magazines to storage arrays, uploading footage to editorial servers, managing backups and so on. Apart from collecting cards from ACs, and delivering them back, data wranglers do most of their work in the camera truck or in an editorial room.
Digital imaging technicians, on the other hand, can usually be found in video village right alongside the cinematographers. DITs are tasked with helping the camera department crew, and the DOP in particular, to get the highest image quality out of the available equipment. This includes using a variety of instruments like waveform monitors and spectroscopes to constantly monitor the camera signals in real time, making sure the settings and exposure are exactly where they should be. A skilled DIT can catch many technical mistakes or problems as they happen, giving the production the chance to correct them on the spot, avoiding costly reshoots or extensive post production work.
As you can imagine, both jobs are very hands-on and require great concentration, as mistakes in ether can be very expensive. This is why these positions are best kept separate.
