An AI system can only be as good as the data it learns from. Before you choose a model, you need to ask a more basic question: do you have the right data, and are you allowed to use it?
Garbage in, garbage out
If the training data is wrong, incomplete or one-sided, the AI system will repeat those problems at scale. This principle is often summarised as "garbage in, garbage out." For most real projects, collecting and preparing good data takes more time than building the model itself.
Types of data
- Structured data fits neatly into rows and columns: sales records, sensor readings, exam scores.
- Unstructured data has no fixed format: text, images, audio and video. Much of the world's data is unstructured, and modern AI is especially good at working with it.
What makes data "good"?
| Quality | Question to ask |
|---|---|
| Accurate | Are the values and labels correct? |
| Complete | Are important values missing? |
| Representative | Does it reflect all the people and situations the system will serve? |
| Timely | Is it recent enough to reflect today's reality? |
| Relevant | Does it actually relate to the problem you want to solve? |
Example
A speech-recognition tool trained mostly on voices from big cities may struggle with rural accents. The data was not representative, so the tool works worse for the people it was not trained on.
Preparing data
- Collect data from reliable, permitted sources.
- Clean it: remove duplicates, fix errors, handle missing values.
- Label it where needed, ideally checked by more than one person.
- Split it into training data (to learn), validation data (to tune) and test data (to measure fairly on unseen examples).
Privacy and consent
Data about people is personal data. Using it responsibly means collecting only what you need, explaining why you collect it, getting consent where required, keeping it secure and deleting it when it is no longer needed. In India, the Digital Personal Data Protection Act, 2023 sets legal duties for organisations that process personal data. If your idea depends on personal data, explain how you would protect it.
Key takeaways
- Data quality decides AI quality: garbage in, garbage out.
- Good data is accurate, complete, representative, timely and relevant.
- Split data into training, validation and test sets.
- Treat personal data with care and respect privacy laws.