Hi,
I would think that you need to improve your actual data model. The reasons are the following:
- Time. Of course the first problem to fix is the load time.
- PSA. It’s not the correct place to store data. You may lose data.
- The logic of count and store the last date can be faced with the natural behavior of a DSO load.
My recommendation is to create a New DSO with the same layout as your datasource and without any modification. This is where you will store all data obtained from your source system.
So this is what I recommend.
A. Actual. SourceSys - PSA – Transf ( Logic) - DSO 1
B. New. SourceSys - PSA – NEW DSO – Transf ( Logic) – DSO 1
In this new model, NEW DSO should has the same KEY’s as the PSA and it should receive all 30 million just one time.
Then to replicate the following Logic:
Source PSA
COMPID DATE TIME and some other fields.
101 28/03
101 29/03
102 30/03
Target DSO
COMPID DATE TIME COUNT.
101 29/03(latest date) 2
102 30/03 1
You can do this:
In the DSO 1
- COMPID should be the only KEY,
- DATE/TIME should not be part of the key (the rule should be as replace)
- The COUNT should be read a COUNT from New DSO and the rule should be ADD.
Whit a DELTA load from NEW DSO to DSO 1, you will only add new records and avoid load 30 million each time.
By doing this, every load will add a count for a new record and will replace the last day for any COMPID. I would think that this new process, should run within minutes in a new process chain.
Hope this may help you.