RAWcooked

RAWcooked easily encodes RAW audio-visual sequences into a lossless video stream, reducing the file size by between one and two thirds. FFmpeg encodes the audio-visual data into a Matroska container (MKV) using the video codec FFV1, and audio codec FLAC. The metadata accompanying the RAW data is fully preserved, along with additional sidecar files such as MD5 checksums, LUT or XML if desired. This allows for the management of these audio-visual file formats in an effective and transparent way. The lossless Matroska audio and video streams can be played in VLC or MPV media players, and writing and retrieving from storage devices such as LTO is significantly quicker.
If you need to use the RAW source in its original form, one line of code will easily restore it bit-by-bit, faster than retrieving the same file from LTO tape storage.


RAWcooked uses FFmpeg for the encoding, see how to install FFmpeg.

The challenges of archival storage

Many archives preserve their scanned films as TIFF or DPX image sequences. These are grouped as multiple film reels, or individual reels in folders that are often TAR or ZIP wrapped before being copied to longterm storage. Since the beginning of digital moving image preservation it has been the dream of many archives to have a single file format that is suitable for the preservation of both film and video.

For further context:

The missing piece of software

The solution MediaArea have created to meet this need is RAWcooked. This software encodes from, and decodes to, «raw» audio-visual image sequences in a revolutionary and transparent way. The image and/or sound content, including all the embedded metadata and sidecar files, are fully preserved at the bit level.
An otherwise unviewable image sequence can be viewed easily with VLC or MPV when encoded as a Matroska file. Plus this process is fully reversible returning your image sequence back to its original package content.

Matroska file complete reversibility

Whenever necessary, RAWcooked can decode the Matroska file back to the original RAW image sequence, including restoration of the original metadata and sidecar files. It is important to stress that the encoded files can be fully decoded, and that this process creates files which are bit-by-bit identical to the originals. Not only is the image and/or sound content fully preserved, but also all enclosed metadata and all of the file’s characteristics. Therefore, an encoded and decoded RAW file cannot be differentiated from its original.

A complete ecosystem

The Matroska files that have been encoded by RAWcooked can be used with many popular open source softwares which support FFV1 and FLAC playback, including:

  • VLC media player for quick and simple review
  • FFmpeg for transcoding, playback, metadata viewing and extraction
  • QCTools advanced quality control software for audio-visual files
  • MediaInfo display and export technical metadata
  • MediaConch for checking the conformance of a file

Being open source, they’re low cost (and free for the many commonly used flavors) allowing archives with limited resources to process the files encoded by RAWcooked on any computer, such as for cataloging or access purposes.

Status

RAWcooked is ready for production. As with any other tool, keep doing basic checks, e.g. testing reversibility with your own files.

rawcooked --all /path/to/sequence/
Encodes the image sequence and the audio files of the folder into sequence.mkv, stores the hashes of the source files in it and checks that the Matroska file can be decoded back to the source files. The source files are left intact.
rawcooked sequence.mkv
Decodes the Matroska file back to the original files (image sequence, audio files, sidecar files), bit-by-bit, and checks them against the stored hashes.
rawcooked --check sequence.mkv
Checks that the Matroska file would be decoded to the original files, without writing them.

More in the user manual, in the man page (rawcooked --help) and in the case study of the BFI National Archive.

FFmpeg

RAWcooked needs FFmpeg for the encoding. Any FFmpeg with the FFV1 and FLAC encoders may be used if ffmpeg is in the PATH (e.g. the FFmpeg of your Linux distribution or of Homebrew), and the -b option selects another FFmpeg binary.

new We recommend the FFmpeg builds of MediaArea: they have GPU support and RAWcooked finds them automatically. Their libraries are also used by RAWcooked for decoding with the GPU.

  • Windows: create a ffmpeg directory beside rawcooked.exe and put the content of FFmpeg_Bin_Latest_Windows_Shared_x64.zip in it (ffmpeg.exe is then in ffmpeg\bin). The Static package has no library, so no decoding with the GPU. The GPU support uses the Vulkan driver installed with the driver of the graphics card (NVIDIA, AMD or Intel).
  • macOS: create a ffmpeg directory beside rawcooked and put the content of FFmpeg_Bin_Latest_Mac_Shared_universal.zip in it (ffmpeg is then in ffmpeg/bin), or run the installer of ffmpeg-ma_Latest_Mac.dmg. The GPU support needs a Mac with Apple silicon and macOS 26 or later, the Vulkan driver is in both packages.
  • Linux: install the ffmpeg-ma package of your distribution from the list of builds, it does not replace the FFmpeg of the distribution. The GPU support uses the Vulkan driver of the graphics card (driver of NVIDIA, or Mesa for AMD and Intel).

More details are in the README, and --show-stats shows if the GPU is used.

Supported input formats

D = Supported by default K = Supported with a key, please contact us N = Need file & development if interest, please contact us

A key for a flavor is needed only for encoding: decoding a file back to the original files and checking it never need a key.

DPX

Uncompressed, 1 image element, unsigned samples, left to right and top to bottom.

Bit depth Packing Endian­ness Y RGB RGBA
8-bit K D K
10-bit Filled A LE N D K
BE K D K
Filled B LE N N N
BE K N N
12-bit Packed LE N N N
BE K K K
Filled A LE N K K
BE N K K
16-bit LE K K K
BE K K K
16-bit float N N N
  • 8-bit: files of both byte orders.
  • 12-bit packed big endian (RGB, Y): also bottom to top.
  • Y 10-bit: also the files without padding at the end of the lines of Lasergraphics scanners (Director included new), DIAMANT-Film and Scanity.
  • Padding bits which are not zero are kept (--check-padding).
  • Some common non conformities are accepted, e.g. a version number in lowercase or empty, the packing value 3 with Y 16-bit.

TIFF

Uncompressed, strips stored one after the other, unsigned integer samples, top left orientation.

Bits per sample Y RGB RGBA
Chunky Planar
2-bit K new K new K new N
4-bit K new K new K new N
6-bit K new K new K new N
8-bit K K K new K
10-bit K new K new K new N
12-bit K new K new K new N
14-bit K new K new K new N
16-bit LE K K K new K
16-bit BE K K K new N
  • 2 to 14-bit: files of both byte orders (II and MM); the samples of 2, 4, 6, 10, 12 and 14 bits are packed as in the TIFF specification, the padding bits at the end of the lines are kept if they are not zero.
  • RGBA: unassociated alpha.
  • XMP, IPTC, Photoshop, Exif, GPS, ICC profile and PrintIM tags are accepted new.
  • The new flavors need a recent FFmpeg, and --check is always done for them.

EXR

Uncompressed scan lines in increasing Y order, single part, data window same as the display window.

Components Pixel type Support
RGB 16-bit float (half) K
32-bit float N
32-bit unsigned integer N
  • Common attributes and the attributes of ARRI are accepted, as well as the attributes of FilmLight and DPX metadata new.
  • The support of EXR depends on the FFmpeg version, --check is required.

AVI

Video Audio Support
v210 (YUV 4:2:2 10-bit) None, or PCM (same flavors as WAV) K
  • 1 video track, then 1 audio track at most; width multiple of 48 pixels.
  • --check is always done.

Audio

PCM in Sampling rate Bit depth Channels Support
WAV, BWF, RF64 48 kHz 16-bit 2 D
44.1, 48, 96 kHz 8, 16, 24, 32-bit, 32-bit float 1, 2, 4, 6, 8 K
AIFF, AIFF-C 44.1, 48, 96 kHz 8, 16, 24, 32-bit 1, 2, 4, 6, 8 K
Other sampling rates, other counts of channels N
  • AIFF: big endian, and little endian (sowt) with 16-bit; 8-bit signed or unsigned.
  • 32-bit audio is stored without compression in the Matroska file.

Another format/flavor you want us to support (e.g. little endian…)? Contact us.

Supported output formats and features

Container format Stream format Support
Matroska (MKV) FFV1 D
FLAC D
PCM (-c:a copy) K
Feature Support
Decoding, checks of reversibility, hashes, conformance and coherency checks, 2 tracks (e.g. 1 video and 1 audio) D
More than 2 tracks (e.g. several reels or several audio files) K
Encoding and decoding by the GPU K new
Encoding options (-coder, -context, -g, -level, -slicecrc, -slices), input options (-framerate), general options (--attachment-max-size, --display-command, -threads) K
Keys for sub-licensees, e.g. your suppliers (--sublicense) K

Another format you want us to support (e.g. MOV…)? Contact us.

Funding plan

MediaArea.net has achieved implementation of RAWcooked’s basic module with thanks to AV Preservation reto.ch, who has guaranteed software development with funding of 10 000 €.

The archive community or interested industry users are invited to buy additional flavors required by their institution - such as components, bit depths, endianness, packing, etc. in order to help the long-term sustainability of the project and its further development. Required flavors will be fully tested by MediaArea.net to exact specification before being implemented into the software.

This model allows contributors the assurance that their files will work seamlessly with the software from the outset.

The price is 1000 € for basic support including bug fixes and one additional flavor, then 500 € for each following flavor which can be added later if you are not sure of how many you need immediately.

The key is stored with rawcooked --store-license key, and rawcooked --show-license lists the licensed flavors and features.

License:

It is Open-Source software which means that end users and developers have the freedom to study, improve and redistribute the program (BSD 2-Clause license for the code developed by MediaArea).