DATA LAKE
READ MORE
DATA LAKE
Up until a few years ago companies relied solely upon data warehouses for data
storage. However, with the explosion of data and the need to support the
increasingly larger volumes, data lakes have become a more practical solution.
Data warehouses were being used to implement traditional processes (ETL) that
require characterization, modeling, and development which created bottlenecks
that delayed companies’ access to information.
A data lake answers this problem by eliminating the need for time consuming ETL. It
serves as a central repository and the the source to which all organizational data
come from. This is often based on Hadoop technology but other cloud solutions can
be used. The traditional data warehouse continues to serve a critical role, it just
evolved to another level in data processing
campaign.
THE CHALLENGE
The challenges in developing a data lake are many so we won’t go into the whole list here. The guiding principles are:
- There should be no size limitation as a working assumption
- There are no limits on processing capabilities (this is a function of applied resources)
- Preventing the lake from becoming a “swamp” requires close management of what is included, documenting the source of the data elements, the frequency of the update and other characteristics (read more the in the Data Catalog section).
WHY DATAMAX?
Datamax has extensive experience in creating data lakes. Our vast skillset and industry experience ensures we deliver robust solutions to our customers in the region.