tf.data.experimental.AutoShardPolicy
Stay organized with collections
Save and categorize content based on your preferences.
Represents the type of auto-sharding to use.
OFF: No sharding will be performed.
AUTO: Attempts FILE-based sharding, falling back to DATA-based sharding.
FILE: Shards by input files (i.e. each worker will get a set of files to
process). When this option is selected, make sure that there is at least as
many files as workers. If there are fewer input files than workers, a runtime
error will be raised.
DATA: Shards by elements produced by the dataset. Each worker will process the
whole dataset and discard the portion that is not for itself. Note that for
this mode to correctly partitions the dataset elements, the dataset needs to
produce elements in a deterministic order.
HINT: Looks for the presence of shard(SHARD_HINT, ...) which is treated as a
placeholder to replace with shard(num_workers, worker_index).
Class Variables |
|---|
|
AUTO
|
<AutoShardPolicy.AUTO: 0>
|
|
DATA
|
<AutoShardPolicy.DATA: 2>
|
|
FILE
|
<AutoShardPolicy.FILE: 1>
|
|
HINT
|
<AutoShardPolicy.HINT: 3>
|
|
OFF
|
<AutoShardPolicy.OFF: -1>
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2024-04-26 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-04-26 UTC."],[],[],null,["# tf.data.experimental.AutoShardPolicy\n\n\u003cbr /\u003e\n\n|-------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.16.1/tensorflow/python/data/ops/options.py#L88-L152) |\n\nRepresents the type of auto-sharding to use.\n\n#### View aliases\n\n\n**Compat aliases for migration**\n\nSee\n[Migration guide](https://www.tensorflow.org/guide/migrate) for\nmore details.\n\n[`tf.compat.v1.data.experimental.AutoShardPolicy`](https://www.tensorflow.org/api_docs/python/tf/data/experimental/AutoShardPolicy)\n\n\u003cbr /\u003e\n\nOFF: No sharding will be performed.\n\nAUTO: Attempts FILE-based sharding, falling back to DATA-based sharding.\n\nFILE: Shards by input files (i.e. each worker will get a set of files to\nprocess). When this option is selected, make sure that there is at least as\nmany files as workers. If there are fewer input files than workers, a runtime\nerror will be raised.\n\nDATA: Shards by elements produced by the dataset. Each worker will process the\nwhole dataset and discard the portion that is not for itself. Note that for\nthis mode to correctly partitions the dataset elements, the dataset needs to\nproduce elements in a deterministic order.\n\nHINT: Looks for the presence of `shard(SHARD_HINT, ...)` which is treated as a\nplaceholder to replace with `shard(num_workers, worker_index)`.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Class Variables --------------- ||\n|------|-----------------------------|\n| AUTO | `\u003cAutoShardPolicy.AUTO: 0\u003e` |\n| DATA | `\u003cAutoShardPolicy.DATA: 2\u003e` |\n| FILE | `\u003cAutoShardPolicy.FILE: 1\u003e` |\n| HINT | `\u003cAutoShardPolicy.HINT: 3\u003e` |\n| OFF | `\u003cAutoShardPolicy.OFF: -1\u003e` |\n\n\u003cbr /\u003e"]]
View source on GitHub