Data.Config

Component: Data

class Data.Config

Bases: Component.Config

All Attributes (including base classes)

source: DataSource.Config = TSVDataSource.Config()

Specify where training/test/eval data come from. The default value will not provide any data.

batcher: Batcher.Config = PoolingBatcher.Config()

How training examples are split into batches for the optimizer.

sort_key: Optional[str] = None

in_memory: Optional[bool] = True

cache numberized result in memory, turn off when CPU memory bound.

Subclasses

• PackedLMData.Config

Default JSON

{
    "source": {
        "TSVDataSource": {
            "column_mapping": {},
            "train_filename": null,
            "test_filename": null,
            "eval_filename": null,
            "field_names": null,
            "delimiter": "\t",
            "quoted": false,
            "drop_incomplete_rows": false
        }
    },
    "batcher": {
        "PoolingBatcher": {
            "train_batch_size": 16,
            "eval_batch_size": 16,
            "test_batch_size": 16,
            "pool_num_batches": 1000,
            "num_shuffled_pools": 1
        }
    },
    "sort_key": null,
    "in_memory": true
}