ffcv.fields module

Fields define the type and the storage method of each of the attributes of a training sample. See the Writing a dataset to FFCV format and Making an FFCV dataloader guides for information on usage.

class ffcv.fields.Field

Abstract Base Class for implementing fields (e.g., images, integers).

Each dataset entry is comprised of one or more fields (for example, standard object detection datasets may have one field for images, and one for bounding boxes). Fields are responsible for implementing encode and get_decoder_class functions that determine how they will be written and read from the dataset file, respectively.

Note

It is possible to have multiple potential decoder for a given Field. RGBImageField one example. Users can specify which decoder to use in the piplines argument of the Loader class.

See here for information on how to implement a subclass of Field.

abstract property metadata_type: dtype

abstract static from_binary(binary: ARG_TYPE) → Field

abstract to_binary() → ARG_TYPE

abstract encode(metadata_destination, malloc)

abstract get_decoder_class() → Type[Operation]

class ffcv.fields.BytesField

A subclass of Field supporting variable-length byte arrays.

Intended for use with data such as text or raw data which may not have a fixed size. Data is written sequentially while saving pointers and read by pointer lookup.

The writer expects to be passed a 1D uint8 numpy array of variable length for each sample.

property metadata_type: dtype

static from_binary(binary: dtype([('f0', 'u1', (1024,))])) → Field

to_binary()

encode(destination, field, malloc)

get_decoder_class() → Type[Operation]

class ffcv.fields.IntField

A subclass of Field supporting (scalar) integer values.

property metadata_type: dtype

static from_binary(binary: dtype([('f0', 'u1', (1024,))])) → Field

to_binary()

encode(destination, field, malloc)

get_decoder_class() → Type[Operation]

class ffcv.fields.FloatField

A subclass of Field supporting (scalar) floating-point (float64) values.

property metadata_type: dtype

static from_binary(binary: dtype([('f0', 'u1', (1024,))])) → Field

to_binary()

encode(destination, field, malloc)

get_decoder_class() → Type[Operation]

class ffcv.fields.RGBImageField(write_mode='raw', max_resolution: Optional[int] = None, smart_threshold: Optional[int] = None, jpeg_quality: int = 90, compress_probability: float = 0.5)

A subclass of Field supporting RGB image data.

Parameters:

• write_mode (str, optional) – How to write the image data to the dataset file. Should be either ‘raw’ (uint8 pixel values), ‘jpg’ (compress to JPEG format), ‘smart’ (decide between saving pixel values and JPEG compressing based on image size), and ‘proportion’ (JPEG compress a random subset of the data with size specified by the compress_probability argument). By default: ‘raw’.

• max_resolution (int, optional) – If specified, will resize images to have maximum side length equal to this value before saving, by default None

• smart_threshold (int, optional) – When write_mode=’smart, will compress an image if it would take more than smart_threshold times to use RAW instead of jpeg.

• jpeg_quality (int, optional) – The quality parameter for JPEG encoding (ignored for write_mode='raw'), by default 90

• compress_probability (float, optional) – Ignored unless write_mode='proportion'; in the latter case it is the probability with which image is JPEG-compressed, by default 0.5.

property metadata_type: dtype

get_decoder_class() → Type[Operation]

static from_binary(binary: dtype([('f0', 'u1', (1024,))])) → Field

to_binary()

encode(destination, image, malloc)

class ffcv.fields.NDArrayField(dtype: dtype, shape: Tuple[int, ...])

A subclass of Field supporting multi-dimensional fixed size matrices of any numpy type.

property metadata_type: dtype

static from_binary(binary: dtype([('f0', 'u1', (1024,))])) → Field

to_binary()

encode(destination, field, malloc)

get_decoder_class() → Type[Operation]

class ffcv.fields.JSONField

A subclass of BytesField that encodes JSON data.

The writer expects to be passed a dict that is compatible with the JSON specification.

Warning

Because FFCV is based on tensors/ndarrays the reader and therefore the loader can’t give return JSON to the user. This is why we provide unpack which does the conversion. It’s up to the user to call it in the main body of the loop

property metadata_type: dtype

encode(destination, field, malloc)

static unpack(batch)

Convert back the output of a JSONField field produced by Loader into an actual JSON.

It works both on an entire batch and will return an array of python dicts or a single sample and will simply return a dict.

class ffcv.fields.TorchTensorField(dtype: torch.dtype, shape: Tuple[int, ...])

A subclass of Field supporting multi-dimensional fixed size matrices of any torch type.

encode(destination, field, malloc)