Data engineering needs to be taken into account while creating a global data architecture.

People frequently believe that the process of Bentley Microstation collecting, verifying, standardizing, and storing geographical data is data engineering. These procedures nonetheless differ somewhat from "data engineering" as it is known in the world of mainstream IT. These procedures nevertheless differ from "data engineering" as it is known in modern IT.

The definition of "data engineering" as used in mainstream IT is: "Engineering is an important content, core, and CDE Solution provider trend of computer science and technology disciplines. It serves as the fundamental starting point for data processing, analysis, and application methods and technologies."

The term "engineering" is highlighted specifically in this description. The goal of "engineering" is to find solutions to problems, to realize value, to be limited to particular business scenarios, to typically need to be taken into account in a comprehensive trade-offs, and to have a strong practice. This requires interacting with users repeatedly in "service" mode as opposed to market-oriented "product" mode.

Do we initially require a global data architecture?

Many would assert that "we just do data processing, data migration, etc., do not need data architecture", "we just do data analysis and presentation, other things do not need to consider so much".........

You can state that if you are implementing a project locally and CDE solution are just concerned with the local scope and particular needs of a particular data processing task. However, you must take data engineering into account from a global viewpoint if you are looking at the project from a broad perspective or if it is greater in scope. If not, a number of issues will develop. For instance, if the statistics in the data are accurate and reliable, whether "data silos" are created, and so on.

Second, when should we start thinking about global data architecture?

An alternative, more prevalent opinion is that "we only do business systems, do not consider analytic applications for the time being, in the future to build business intelligence (BI), data warehousing applications, we will then consider the data architecture."

Whether or not there is a distinct data architecture may not matter much if there are only a few business systems. But if there are more than five business systems, this "system over data" and "process over analysis" mindset would cause a lot of issues. Without a unified data architecture and data governance mechanism, there will be disparate data standards, disparate data content, data with the same name but different meanings, data quality cannot be guaranteed, and data integration is very challenging, all of which will adversely affect the business application system's regular performance. Even if the data extraction, processing, analysis, and presentation tools are effective, they are useless if the business system as a whole has issues with data quality. The global data architecture must thus be taken into account for big, sophisticated business application systems; in contrast, it will be challenging to advance data analysis applications without a data architecture and data governance processes.

How should global data architecture be done, third?

The goal of global data architecture is to provide a solution to the user's organizing dilemma so they may effectively manage and utilize their data assets. Multiple factors, including the data resource catalog, data standards, data model, data distribution, and others, might be taken into account as an answer to this issue. Metadata management, data integration, data exchange, and other factors should be taken into account for the unique landing. Our overall data architecture design considerations for a particular project are shown in the following figure.

Logic chart for a project's global data architecture

The data is separated into various libraries based on the application orientation from a broad perspective:

Business Library

It is mostly focused on "business processors" in the "business application domain". From the perspective of data, a library contains multiple data domains, and its relative, a platform for numerous applications, or a business platform to support numerous business applications, constitutes the entire "business application domain" as a system of a library. This system's primary purpose is to fundamentally address the issue of the previous dozen or so systems, which led to a "chimney system" problem. Additionally, the business library of the data organization form is intended to "handle matters" for the organization of data modeling; data operations often involve additions, deletions, updates, and checks; these activities are characteristic of an OLTP (online ledger transaction processing) database.

Analytical library

mostly for "analysis of decision makers" in the "data analysis domain." Hence, the requirement for creating a data warehouse. Operational data warehouses (ODS), core data warehouses (DW), data marts (DM), etc., are examples of data warehouses built in accordance with various application scenarios. These data warehouses are also used to develop the appropriate "data application platform" and a number of data applications. The "analysis theme" dictates how the data are arranged in the analysis library. The so-called "analysis theme" refers to the analysis requirements for a certain business product or subject, such as the distribution analysis, housing mobilization analysis, or analysis of building projects.

Public library

The "data governance domain" is mostly focused on "data governance personnel," as the name would imply. All of the data for the whole agency is managed through data governance. Among these, "master data" is the information arranged in accordance with "core business objects" that offers the shared core data base plate and has the properties of unity, completeness, correctness, and timeliness. For instance, housing listings are a sort of master data in the domain of public housing. The information used to describe the data, such as its kind, linkages, flows, modifications (lineage), and business consequences, is referred to as "metadata." "Reference data" refers to a few key data dictionaries, such as those used in the field of public housing to represent terms like lease status, reason for registering to leave, method of rent payment, and housing status.

Additional libraries

There are some more data in addition to the aforementioned core library. Along with different document materials, electronic files, and other unstructured data, exchange data is utilized for both internal and external data exchange. geographical data is used for positioning and geographical analysis.

Finding out what data are available globally is the global data logic architecture's greatest benefit. How are different forms of data related to one another as well as to systems? How are the many types of data managed and stored? Data model, which describes the precise relationship between the data from a static perspective and directs the logical design of the database and physical design that follows; data distribution, which describes the distribution of data in the business application system from a dynamic perspective, a broad view of the flow of data; and so forth are also included in the data architecture. Not featured here due of space restrictions.

CDE Solution provider

50