+1 vote
Good day!

Faced with the necessity of collecting data from multiple systems, for further use with the purpose of generating analytical reports.
Powerful on the Internet, but anything intelligible did not find, so ask for help.

Are in search of solutions able to meet the following requirements:
-Collecting data from multiple databases from different vendors, oracle db, mssql, etc.
-Configuring time ranges collect (every day, every hour, etc.)
-Select full or incremental fee depending on the conditions (such as once a month full and every hour incremental)
-Setting tables and fields from which data is collected (for each system its own set)
-Availability of remote access to systems from which data can be collected (e.g. data warehouse in St. Petersburg, and the target system in Moscow and Novosibirsk)
-Zoom (new systems)

The area is new to me, if you need clarification, I am ready to answer all questions.

Thank you.

1 Answer

+1 vote
Thank you for inviting me as an expert.
I have to say that the solution in the General case.
You will have to create an aggregator of the data independently.

Option 1. A very smart application. Mega Crawler.
App Mega Crawler must support the ability to establish remote connections to multiple database types. The app needs to know about the data structure of each database to which it clings.
It will be either hard-coded logic to change only the programmer.
Or configurable logic, the development of which can leave the basic resources of development and which will be powerless when connecting to N+1 database.
The most known analogue is the yandexbot. Tries to parse the page, find lyrics, build on them index.

Option 2. Application network agents. Spider Net.
Spider is required data from one database and forwarding the data to the center "web" using the specified Protocol and in a format which realizes the Central control node.
The most known analog - Yandex.Market. Online shopping themselves realize the unloading of goods and transfers to the Yandex.Market in a pre-specified format.

And what options do you have?
Yes, thank you very much. for Your answer!