Article

Use SageMaker AI to check a dataset before training

For AI teams preparing text data, choose duplicate, label and task checks, review their limits, and budget for SageMaker compute and model usage.

Editorial illustration for Use SageMaker AI to check a dataset before training: a geometric block represents a model release. Not documentary evidence.

AWS made SageMaker AI’s data preparation tools generally available on November 17, 2026, giving AI teams a managed way to inspect the examples they use to train or test models. For an existing text dataset, register the data, select checks for the relevant columns, then review the reports before making changes.

Review flagged records before changing the dataset. The quality-evaluation job leaves the input unchanged, and some checks examine only a sample. A completed run is not a certificate that every record is suitable for training.

Check the region and register your dataset

Quality jobs run in US East (N. Virginia) and US West (Oregon). Every input and output S3 location must be in the same region as the job. AWS’s supported-region list does not include an India region for this workflow. If your data must stay elsewhere, resolve that requirement before uploading it. The service supports text, not images, audio or video.

For the SDK route, use Python 3.10 or later and SageMaker Python SDK 3.30.0 or later within version 3. AWS’s setup guide also calls for upgrading boto3 so the underlying API definitions are current. Configure an execution role and an S3 output bucket. Check both the submitting identity’s permissions and the role’s S3, hub and Bedrock access; these are separate in the quality-job permissions guide. A coding assistant is optional.

Upload the records to S3 and register them in a SageMaker hub with DataSet.create(). Keep the returned ARN—the dataset identifier used by subsequent jobs. The registration instructions require validation_role_arn when you declare a format other than Custom. Custom skips format validation; it does not mean the data has passed a quality check.

Select checks for the columns that matter

Start with the problem you want to find. AWS’s operator catalogue separates the following checks. In each operation, Config.View identifies the column and its role, such as Text or Label.

  • Repeated text: select exact_dedup or fuzzy_dedup on the relevant text column. These flag matching or similar text and identify a representative row; neither removes records.

  • Uneven labels: select label-balance metrics such as data_balance on your label columns. Use the result to inspect missing or overrepresented categories, not as proof that the labels are correct.

  • Task-specific quality: use spec_eval to assess whole rows against rules you write, with reference documents where needed.

  • Personal information: add pii_detection explicitly, naming one Text or Messages column. This is a sampled check for direct identifiers, not a scan of every record.

Do not treat default_metrics as an all-checks switch. Its nine text-quality measurements do not include the duplicate, label-balance or PII operations above. The practical consequence is that a successful default run can leave your main question unanswered.

The PII documentation does not specify the sample size in its operator description. It also excludes information that identifies someone only when combined with other information. A sample with no detected personal information cannot establish that the entire dataset is clear.

Write rules that reflect the intended task

A useful rule states what makes an example acceptable. As a hypothetical support-data rule, you might require a proposed reply to answer the customer’s question without contradicting the supplied refund policy. That is more specific than asking whether the reply is “high quality”; it is an illustrative criterion, not a tested rubric.

Put the rules in a prose or structured specification file and supply its S3 location through SpecS3Uri. For document-grounded judgments, use ContextDocumentsS3Uri. Review those files before submission: AWS says instructions embedded in specifications and reference documents are passed to the judging model without being removed.

Use DownSampleRecordCount for an initial specification check, then inspect the derived criteria and judgments against examples whose expected answers your team knows. Rows outside that sample are unjudged, not passing. Reviewing the rubric and expanding row coverage are separate decisions.

For synthetic training tasks, also check that the expected outcome is actually verifiable. RohitAI’s AutoSynthData guide to validating generated training tasks explores that upstream question; it is not a SageMaker integration.

Budget before submitting a larger job

The AI data preparation pricing section separates three relevant charges: compute for every job, service input/output tokens for free-form quality analysis, and Bedrock usage when the job invokes models. Compute is billed by the second with a one-minute minimum, on an instance size selected by SageMaker. No service-token charge does not mean a free job.

AWS’s worked example for free-form analysis of 10,000 rows totals $1.1266: roughly $0.1723 for compute and $0.9543 for service tokens, before separately billed Bedrock calls. Those are AWS’s example costs, not a quote for your dataset or a fixed price per row. Text length, selected checks and model usage affect the bill.

For spec_eval, AWS documents a model call to derive the criteria, followed by calls for row-level criteria that require a model. Code-decided criteria need no per-row model calls. Use the job’s token_usage_report.json and the selected models’ current prices to refine your estimate; do not assume that choosing simple rules eliminates every model charge.

Run the job and inspect the report files

Use DatasetJobConfig.print_schema() for the data-quality job category to inspect the service’s current fields. Build the configuration with the registered dataset ARN, output S3 path and your selected Operations. The current SDK example explicitly requires schema_version="3.0.0" for this request structure. Submit it with DataPreparer.evaluate() and retain the job ARN.

Check size limits first: a job containing exploratory analysis allows up to 18 GB; one containing spec_eval allows up to 30 GB; other quality jobs allow up to 90 GB. A combined job takes the smallest applicable limit. The documented default quota is three concurrent quality jobs per account per region, adjustable by request. Bedrock quotas apply separately.

After completion, find MetadataS3Uri in the DescribeJob response or dataset entry. It points to metadata.json, which locates the reports and judging artifacts. For the checks above, inspect:

  • aggregate/quality_report.json: dataset-level measurements and rule summaries.

  • record_level/evaluated_data.jsonl: the original records with result columns added. Review duplicate groups, failed criteria and criteria that could not be evaluated.

  • spec_eval/: the derived criteria, judge code and configuration. Compare these with the rules you intended.

  • token_usage_report.json: model usage, located separately in the output folder rather than listed by the metadata index.

Treat null specification results as “not judged.” Review both passing and failing examples before using a pass rate as an acceptance threshold. The important distinction is between how many records received a judgment and whether those judgments match your task.

Make changes separately, then check the revised data

Do not automatically discard every repeated answer or force every label to occur equally often. As a practical judgment, the same reply may fit different questions, while an intentionally uneven dataset may reflect the intended use. Have the task owner review the affected records and decide what should change.

When a change is justified, data transformation is a separate job. Most transformations publish a new dataset version while preserving the input; each transformation job takes one operator. Re-evaluate the revised version and retain its dataset identifier, chosen columns, rules and reports together.

Finally, test the resulting model on the actual task. Dataset inspection and model evaluation answer different questions: our StateBench explainer on tracking changes in long documents illustrates why apparently readable input can still demand a difficult reasoning task. Use this workflow to identify and review data problems, not to substitute a quality score for evidence that the model performs the job.