Media Management 101: Basics of Data Wrangling

With digital cinema showing no signs of slowing downand data rates accelerating – having the skill in your repertoire to properly manage data is crucial. Even if you never do data loading yourself, knowing how the process works helps you manage your department.

In this video I give a brief overview of my basic media management toolkit, how to set up an organized folder system on the hard drives, and a process to make sure you never accidentally erase footage (hint: green tape = good).

Also, this 2nd video below explains in more detail a quick and simple way to check that the footage has been downloaded and backed up safely.

Best viewed in full-screen mode so you can read the on-screen text. You can also watch this on Vimeo.

Data Wrangler Basic Toolkit

In the video I give a brief overview of the tookit I use and mention some items specifically. Here is a list of equipment you see in the video:

In no way is this a super-professional high-speed kit, but it’s good enough to perform basic data wrangling tasks. If you’re a camera assistant consistently working on digital shoots, it’s a good idea to invest in some of these items.

One note about hard drives: it is standard for the production to buy them. Don’t ever use your own money or money they are paying you to buy the drives. Once the footage is on them, the drives will be filtered throughout the post-production pipeline. You are more than encouraged, however, to recommend a few hard drive options.

Video Transcript

Hi everybody this is Evan Luzi from The Black and Blue and today I’m sitting in a hotel room at a Washington, DC hotel doing a little data loading.

Now this isn’t something that I normally do, but every once in awhile a job comes up that, you know, the price is right, the hours are right, so I go ahead and take it.

But I thought it would be useful for you to know sort of the basic data wrangling kit that I have as opposed to somebody who might build a cart worth thousands of dollars. This is sort of the bare bones thing and it’s not the fastest kit in the world, but you might be able to build it by spending a couple hundred extra dollars and it could give you a few jobs every now and then.

So let me just sort of go over what I have going on here. I actually have a dump going on right now of a 32 GB card and it’s been going on for about 40 minutes already — it’s got about 17 minutes left.

We have it dumping to two different hard drives with 2 Terabytes on it each. I’m not sure how much footage they’ve gone through the day yet — I’ve got three 32 gig cards from A Camera and about four from B Camera. And then I had the sound guy’s data, too, to load.

So were dumping it on two backups. That’s pretty standard. The more backups you have, the better, if you can afford three, but two is essential. And one thing an AC taught me that I’ve always remember is always make sure you have two copies of everything before you willingly erase anything.

So don’t format a card until you have two copies, don’t put anything on a new hard drive until you have two copies. Basically you want to have a redundant copy before you move, transfer, modify — whatever you’re gonna do with it — always just have a backup.

So anyway, the main part of my kit besides these hard drives, which production provides — I would never buy my own hard drives, especially because they go to editorial or through the post pipeline when you’re done — but I have an old Macbook Pro. This isn’t one I bought for data loading, it’s just my personal computer. It’s one that I bought in college so it’s kind of old, so I’ve kind of had to beef it up a little bit with peripherals to make it acceptable. Especially with data rates getting so heavy these days.

So just recently I bought a Lexar Expresscard CF Reader for Compact Flash cards and this goes in the little express slot on the side of your Macbook Pro. Newer models, only the 17″ has that expresscard slot, but the newer models also have Thunderbolt which is insanely fast so peripherals will start coming out for that so you can take advantage of that if you aren’t blessed with the expresscard slot.

Also in here I have a similar expresscard adapter for SD cards for some Nikon cameras, most sound guys use SD cards, some of the lower DSLR’s — like the one I’m shooting on right now, the T3i, uses SD cards.

So I have that. Got the cards I’m dumping. I’ve got two rolls of camera tape.

Now this is kind of a different data loading job for me because I’m doing it after they’ve shot everything at the end of the day. Normally I’d be doing it throughout the day.

So what I like to have is green tape that way when I double check the card, when it’s OK to format, I can wrap it in green tape to the point where an AC would have to unwrap it to get to it. They can pop it in the camera and they can format it knowing that the green tape means it has been checked.

I usually go up to the AC or the camera operator or the DP and tell him if you get it without the green tape don’t format it! You know, bring it back to me, it could’ve been my fault but I just want to double check it.

The worst thing that could happen when you’re in this job is having something get accidentally erased because it all comes down to you: there’s nobody else checking footage on set.

So that’s sort of the basic tools I’m using — it’s pretty simple. Most hard drives will come with cables.

Oh — the thing I do have that you can’t see is a battery backup system in case the power goes out and as I’m shooting it’s kind of stormy outside so I’m really glad I have it. It will buy me about 40 minutes worth of time — which, you know, I couldn’t finish the job on that, but I could at least make sure nothing gets damaged. Make sure that the hard drives don’t get damaged if the power goes out.

So the next thing I want to talk about is the layout of the folders that I make on the hard drives to organize the footage.

If you’re doing a multi day shoot, at the most basic level, I just do “Day 1,” “Day 2,” “Day 3,” “Day 4,” etc.

Within that, I’ll make multiple folders. I’ll do an “A Camera” folder, a “B Camera” folder (so on and so forth) then I have a sound folder.

And within those folders, say for the “A Camera” folder, I’ll dump the first — whatever the first card I get — that’s roll “A001,” so I make a folder in that called “A001.”

Then I just transfer everything from the card into that folder — even the thumbnail previews. I don’t even mess with the structure of the card itself, I want to save absolutely everything.

This is integral especially on RED footage where messing with the tiniest little detail could corrupt every little R3D.

So you don’t ever want to just pick and choose which clips you’re downloading, just dump it all into one folder.

So as the day goes on when I get another card from A Camera it will be “A002″ and that goes on. And that carries over over the days. So if at the end of Day 1, I’m on “A008,” at the beginning of Day 2 I’ll be on “A009.”

And so you do that for every camera. There are some days where you only shoot 4 rolls on B Camera, but 10 rolls on A Camera.

Now within the “Sound” folder, most sound guys will have you dump all their stuff at the end of the day or maybe do a dump at lunch and a dump at the end of the day as well. And within those folders, if you want to, you can just do “1″ or “2.” Or you could do like “SND001,” just something to denote that it’s sound.

Every now and then I get on a job where they want me to keep log notes and those I do a Text-Edit document on my Mac. I’ll just save that to whatever day it relates to.

Or if I want to leave note for the editor or whoever is in the post-pipeline — say if a clip is corrupted or if I’m using the labels feature on my Mac — then I’ll leave a little key for them and I’ll type in “Please Read Me” as the title, and I’ll type in whatever notes I have for them.

I tried the route where I’d go up to a producer and be like, “Oh hey make sure you tell him blah blah blah” and it would just get lost in the shuffle. They would either not remember or they’d mess it up with the message and the editor would end up calling me anyway.

It’s a lot easier if you just write everything down into a little text document and save it to where they need to see it. And I think most editors and post people are pretty good. If they see a file they know it’s been put there intentionally and they’re gonna look at it even if they aren’t sure what it is.

So that’s kind of how I do the folder structure. The key for this is you want it to be super organized so when it gets into post-production people aren’t bugging out at how confused they are over which footage is which.

So once I’ve dumped it, I’ll go through and check each folder. And how I do that on a Mac is I hit the apple and the “I” and get the info. I can get the exact number of Gigabytes that a card has on it — you know, how much space is taken up.

Then I check those folders on each drive and make sure that that number is consistent.

And then what I’ll do is I’ll go through and I like to check the first clip on a card, a clip in the middle, and then a clip at the end. And if there’s a bunch of clips — like on this one we have over 200 — I’m probably going to do about 5 – 8 spot checks.

What I do for that is I open it up in Quicktime, I watch the first few seconds, I scrub through the whole thing quickly, watch the end, then it’s good.

Sometimes if you’re on set you may be pressured to be turning cards over very fast, in which case you do have to do the spot checks fast, but please never sacrifice doing your job right for the time.

If you’re getting really pressured, just go up to them and be like, “Look I just need to double check that these cards are safe. I don’t want any footage lost.” People may be frustrated, they may be annoyed, but they’re also going to understand.

Just one thing I want to add to the green tape thing I went over earlier. If you can add a verbal cue, where you say, “This is OK to format,” then I think that is more than enough. People really seem to appreciate when I go up and say, “This is OK to format.”

Because I think whenever somebody formats a card they have that moment of hesitation where it’s “Oh, I hope he checked it,” so I tell them upfront that I did check it. And that little worrisome moment is gone.

So make sure you do say, “it’s OK to format,” though. Don’t be ambiguous — don’t just say “this is good,” “this is OK.”

That could be taken in any way, so just say “it’s OK to format,” make sure it has green tape on it or whatever system you want to do and that’ll be good to go!

So that’s how I do media management. It’s not super complicated — it’s more like copy and paste — it’s to the point where I had the time while this is going on to make this video.

If you can get a job here and there doing this, I would recommend it. It’s simple work, but it usually pays pretty decently and it’s not so bad.

I mean, it could be worse: I could be out hauling gear in the storm or working through the night. Right now I’m in a cushy hotel room that’s 69 degrees just sitting at a computer.

Alright guys well thanks for tuning in. I’m sorry if the sound was a little bit. I’m right off of Dupont Circle in DC, which is a really busy intersection.

Also I’m sorry this video turned out a little bit longer than I hoped. But I hope you enjoyed it nonetheless. See you later!

  • http://www.facebook.com/dustymccord Dusty McCord

    I like to use a program called shot put pro. it will do the copy of source to two locations plus a checksum. which give me an added level of reassurance that all is well.

    • http://www.theblackandblue.com/ Evan

      I have heard about Shotput Pro but have never liked using automated software. Prefer to do it myself. I also don’t get enough data loading work to justify the cost to myself.

  • Cail

    You must, must, must use checksums for this kind of work. Spot checks and scrubbing will not catch small errors.

    R3D Data Manager ( and the Alexa version ) or the new version of Shot Put Pro will do this for you.

    Another thing worth mentioning is that your optimal transfer speed will be reached if you are not daisy-chaining drives. Best to use eSATA drives and FW800 readers (although these are no longer made…)

    • http://www.theblackandblue.com/ Evan

      I disagree that programs like R3D Data Manager and Shotput Pro are a must. I prefer to do manual checksums, spot checks, etc. I used R3D Data Manager for an entire feature film and often it would tell me the checksums weren’t checking out — after multiple transfers — and I would have to do them manually anyway.

      Doing the simple drag-n-drop in Finder, I have never lost one piece of footage ever.

      I don’t disagree that those programs are useful, but I don’t think they are an absolute necessity, especially if you are doing this kind of work once in awhile.

      You’re right about the drives and eSATA and FW800 — an ideal setup would be each drive separately having its own channel into the computer.

      • Cail

        Precisely how are/were you doing manual checksums? Running cmp or md5 in Terminal?

        Point is after 3 features and several TV series, I can count on one hand the number of times CF readers have overheated and corrupted data in a way that visual scanning and file size comparison would miss, but checksums (through any means) would catch.

        So, a small number of occurrences, but anything higher than zero is no good for me and my reputation.

        For me it’s the same line of thinking as making dual copies before wiping the shooting media – the higher quality SSD readers and more robust file formats may help in future but checksums are still part of the process.

        If you were a film loader, would you leave the camera tape off the mag’s seam just because it took a bit of extra time and money? I think not.

        • http://www.theblackandblue.com/ Evan

          I don’t blame you for wanting to be cautious, but I have had less than awesome experiences with checksum software and have had no problems using my method.

          I don’t do data loading or DIT work on a regular basis, instead it’s normally something I default into as an AC. If a production wanted me to or purchased for me either of those programs, I would use them, but it’s not worth it for me to buy them when I haven’t had problems otherwise.

          I think you and I see differently on the process I use. Where you see what I’m doing as a shortcut, I see what I’m doing as an alternative and thus, the camera tape on the mag analogy doesn’t hold water for me.

          It’s all about scalability, too. In this setting, I had no time pressure. If I had a more chaotic production to be working for, I would look into those programs to automate a lot of what I do.

          Different strokes for different folks, and indeed, different environments. 

      • anneso

        Agree. R3D data manger has been a pain for me so far. It would mention ‘Errors’ while it was fine, which caused me stress that I couldn’t afford, so ended up doing an extra back-up for safety so waste of time I dont have really. The only good experience I had with software this kind is SilverStack. But same as you, as a 1st and 2nd AC, I haven’t find the will to spend money into it, while doing it myself has never caused me any trouble so far.

  • Pingback: Basic Data Management With Evan Luzi From The Black & Blue. - Wide Open Camera

  • http://www.facebook.com/dustymccord Dusty McCord

    I feel you on wanting to do stuff manually. I felt the same way. R3D manager is cool because it will do R3D check but shotput pro is basically a gui for rsync that is production centric.  in my opinion its worth the $100.

    its makes a log text file,
    can use manual folder labels or label the folders the same as the drive.
    the biggest thing is the checksum and error reporting.

    in my personal experience this program has found and reported errors that led to me being able to save data that a simple drag and drop could have over looked.

    above all, double checking play back is a must no matter what you do. I actually watch every video on the card before I will delete anything. I have been on jobs where the DP will send me footage they know is bad to see if I will ketch it, to test me.

    • http://www.theblackandblue.com/ Evan

      Hah! Have never heard of purposefully sending bad footage. I’m sure if I ever pursued data management seriously that I would invest in something like SPP, but as an AC, I don’t want to be known as a data manager so I don’t kit myself out for it. It forces producers to find/hire someone else and let me do my AC duties.

      What ultimately made you take the plunge and move away from doing manual checks?

  • Pingback: Basic Data Management With Evan Luzi From The Black & Blue. « Wide Open Camera

  • Gillian Holt

    Hi Evan,
    Firstly a big thank you for a very informative blog.
    My question concerns how you allocate reel numbers. I also have a similar system: I mark a “clean” card with it’s case taped shut with pink camera tape and a “hot” card with the same tape over the card contacts. I pre write these tape markers with the reel number (A001-G) where A is the camera 001 is the reel number and G is the “Mag” or card ident. Each Tag also has a short tear off tab that goes onto the camera so the Op knows which reel is in use. I find this to be a very safe and flexible system but it suffers from the disadvantage that the cards must be used in the right order as the reel numbers come up. I see you send the cards out blank and presumably let the camera Op mark the tags with A001 etc. My question is does this Not potentially lead to someone forgetting which reel the last one was, and is there a third and better way to solve this.

    Best wishes from sunny Berlin.
    Gillian Holt
    Cinematographer/AC/DIT

    • http://www.theblackandblue.com/ Evan

      Hi Gillian,

      Let me speak more from the camera assistants’ perspective, as that’s where most of my experience lies. 

      Generally, I will make a bunch of “Mag tags” out of camera tape with the reel numbers on them at the beginning of the day, or ask my 2nd AC to — “A001″ so on and so forth. Then, each time I remove a card from the camera, I take off the tape from the camera, wrap the card, and have it delivered to the data wrangler/DIT. If the card comes back with the tape on, I refuse to reformat it.

      When I reformat a new card, I ask my 2nd AC for the next mag tag and slap it on. Since I am removing the previous tag, I get a reminder of what reel we were just on, so I don’t have trouble getting mixed up. Also, cameras like RED and Alexa keep track of reel numbers for you, automatically counting up each time you format a new magazine.

      Lastly, usually the script supervisor or 2nd AC is pretty on top of what reel numbers are what with camera reports and continuity notes.

      I like this method because I can format cards in whatever order I want while also keeping the process organized.

      Hope this helps!

      • Gillian Holt

        Thanks Evan,
        I’ll give this a try next time out :)

        All the best
        Gillian.

  • Warren Channell Jr

    Hi Evan,

    Great post, don’t apologize for the length of the video it was very thorough.
    The next time you’re in DC, I owe you a strong drink and a big thanks.

    -Warren in DC

    • http://www.theblackandblue.com/ Evan

      Hey Warren,

      how strong of a drink are we talking? :P

  • Pingback: Discomfort Comes Easy on a Film Set | The Black and Blue

  • http://www.facebook.com/dainis.juraga Dainis Juraga

    Hello!

    Way how I prefer to copy material is “through” laptop internal HDD. First copy from card to internal HDD and just then copy to external drives. Do someone of you have same practice?

    Why? I had once the problem with card reader (FW800) and external USB drive — material copied, but on external HDD material appear drop frames etc. And I think that for power is more safe to copy at first to charged laptop and then to external drive from internal HDD.

    Thanks!
    Dainis

  • Pingback: How to Piss Off Talent by Doing These Three Things Nobody Ever Tells You Not to Do | The Black and Blue

  • Pingback: How to Use a Phantom Miro CineFlash Drive with Mac OS X | The Black and Blue

  • Pingback: Deciphering the Film Slate (Part 1): What to Write on a Clapperboard | The Black and Blue

  • Pingback: Data Management | Tutorials

  • Ben McPhee

    Hey mate, in the 2nd video above, you only made an A001 folder. When would you make an A002 folder? Each time you dumped a card? (So if Camera A needed 5 dumps throughout a day, there would be A001-A005?). And do those numbers correspond to the roll numbers you would have written on the clapper board?

    Also, is it necessary to note anywhere what scene/takes etc were on that card? How would an editor organize this footage, or find a specific scene using this file structure?

    Lastly, this video is a couple of years old now. Is this still how it’s done? Or are there software packages that you can import into, and do spot-checks with? (Are they reliable).

    Thanks. :)

    • http://www.theblackandblue.com/ Evan

      Yes the A001 corresponds to the roll number on the slate. So each time a memory card is “reloaded” into the camera, the roll number increases. Ideally you would have an AC who puts “mag tags” on the card (basically a piece of camera tape with the roll number). I usually make the A00X folders when I begin a new transfer.

      Depending on the shoot, you might make log reports that designate what scene/take corresponds to a file name. Alternatively, some camera reports will have a clip number on them that corresponds to scene/take. It really depends on how much time you have and what the production wants. If you’re doing logging, then you start treading into assistant editor territory.

      As for your last question, it depends. There are a lot of people who advise against simple drag ‘n drop from Finder because it is imperfect. And they’re right. Which is why when I do it this way I make sure to spot-check at least three clips. There is software out there that will do check-sums for you. The most popular is Shotput Pro. From what I’ve heard, it is reliable. The only downside is the price and sometimes these programs can slow down the transfer times as they perform extra checks.

  • http://www.bravocharlie.tv/ Philip Bateman

    Thank you for this – really enjoyed it, I may potentially get a job doing 2nd AC for the first time and your website is a fantastic resource.

    • http://www.theblackandblue.com/ Evan

      Thanks! Glad you enjoyed it Philip!

  • Drew Ruggles

    Wanted to weigh in on the drag and drop discussion here. I dragged and dropped 2 days worth of footage onto a Drobo…got into post and over 50% of the files appear to be “corrupt”… has anyone else experienced this or have a solution/explanation? I’ve included a screenshot of the psychedelic artifacting that seems to have replaced my footage….

  • Ty

    I enjoyed this post. It’s hard to find good resources on something as un-glamerous as data management, but it’s incredibly crucial.

    I’m not complaining here, but would like to suggest: I don’t think I’m alone in saying that a lot of people could benefit from an updated post about data wrangling, especially in the low-medium budget space. I do work at a small agency that specializes in short docs, and we usually run a 3-man crew – a producer, a shooter/director, and a shooter/camera assistant. Sometimes the crew is smaller. Either way, its left to me to do data management at night and more efficiency in management translates directly to precious SLEEP at night. On 3-week shoots where we’re getting just a few hours of sleep a night, another hour of sleep instead of babysitting a transfer is a huge win.

    I’m first to admit that I’m mostly self-taught in this area, but here are a few tips I have picked up from doing this the last 6 years:

    1. You’re only as fast as your slowest link. Make sure all your stuff is USB3.0, Thunderbolt, or eSATA. If you’re using a USB2.0 card reader you are basically doubling your copy times.

    2. Come up with a physical labeling system and stick to it. I like what Evan did, it’s similar to how we do it:
    -Card comes in from shoot, label “NOT COPIED”
    -Card is copied to hard drive #1, “COPIED, NO BACKUP”
    -Transfer all cards to hard drive#1
    -Backup drive #1 to drive #2 and #3
    -Upon successful backup and checksums, label cards “GOOD TO CLEAR”
    -Don’t clear cards on your computer, format them in the camera later just before you use them

    This last point has saved us a time or two. We have a bunch of cards, and try to rotate the ones we’re using, so the cards we shot to 2 days ago are being used before the ones we shot yesterday. In case we realize something has gone terribly wrong, or hard drives get stolen, it insures we might still have some footage.

    3. I have started using a Thunderbolt dock to get more USB 3.0 ports instead of using a hub. This allows me to copy multiple cards to a drive at once, dramatically reducing the time I spend copying. I utilize 3 card readers, plus the SD slot built into my MBP simultaneously.

    NOTE: I would love to ask a real DIT about this, and wether or not this is common practice. We still use spinning hard drives not SSDs, and I wonder if putting the drive under this kind of abuse (writing multiple files to multiple places on the platter) puts the hard drive at greater risk for failure. So far, we have had no problems with this.

    4. I like using bus-powered USB 3.0 drives. It reduces the weight of what you’re carrying, which is always something we’re trying to do with all the airports we’re going through. It’s also nice because if you are in a foreign country, those are a few less power adapters you have to worry about. Furthermore, if you are in a situation where the power isn’t totally reliable, I’ll ditch the Thunderbolt docking station (which must be powered) and just work off the USB prots on my laptop. If the power goes down they will still run off your battery, so you don’t need to travel with a heavy power backup solution.

    Anyways, would love an updated post – I have seen a lot of software for this sort of thing coming out the last couple years, and wonder if anyone is using it.