We then use a view to only show records that don’t have values in the date_end column so we only show current data, not history information.

I posted my thoughts here. Let me know your thoughts.

Niraj

On the other hand (this is the addition to Dalibor’s suggestion), to more efficiently support queries to find when row X was changed, we add a junction table to create a many-to-many relationship between the audit table and the table that stores column metadata (could also be done against the information schema, system views, etc. I guess, though I’ve never done it). This way, when working within a database that does not support XML querying, you can still efficiently trim your results to only changes that affected the value of the particular column you’re working with. If you’re working in a scenario where surrogate primary keys of a uniform data type are used, you can add the primary key as a field to your audit table – at that point it’s easy and fast to get a list of all audit entries that changed the value of a particular cell in your database, regardless of your particular RDBMS. You’ll still have to iterate through the results to find the particular change you’re looking for, but in my experience this is a very efficient and maintainable solution.

Small caveat: creating links between audit rows and column metadata rows can make your auditing model somewhat painful when your schema starts to evolve.

for DDL changs you can use DDL triggers which also give you data in xml format.
with that you can again do the same thing like above.
you can have 2 tables: 1 for DML and 1 for DDL changes.

using this async SB pattern you can have you auditing on its own server that is meant just for auditing.
all child servers can all send audit data to this centra auditeing server.

i’ve implemented this and it works great. also here’s my article on the subject:
http://www.sqlteam.com/article/centralized-asynchronous-auditing-across-instances-and-servers-with-service-broker

Follow Jonathan advice, storing domain events as your main storage, then derive RDBMS views from events. You can easily reconstruct any view of your data from you event stream then.

You get one audit table (which I guess could be filled by some sort of a generic function) with the ability query for the data you would need (via MS SQL xml querying capabilities).

If you are using another database which has no xml querying then I would still go with the same principle, but maybe not with Xml format (maybe ‘column=value’).

100
test
… etc …

You get one audit table (which I guess could be filled by some sort of a generic function) with the ability query for the data you would need (via MS SQL xml querying capabilities).

If you are using another database which has no xml querying then I would still go with the same principle, but maybe not with Xml format (maybe ‘column=value’).

We did encounter an issue when it came to archiving policy. If you we going to say archive everything over 30 days, then it was fine. However, if you wanted to vary the archive terms by entity it was a problem since the logging for all entities was in a single table. If you need to vary archival term by entity, you need to have separate auditing tables for each entity or duplicates of each table for auditing. I suppose you could manually extract audit entries by entity type manually, but that’s error prone.

I’ve compiled some resources on his patterns on my blog:
http://jonathan-oliver.blogspot.com/2009/03/dddd-and-cqs-getting-started.html

Figure out what “Y” is first. Figuring out “Y” is much harder than figuring out how to store the data. You are (probably) thinking about storing data till the end of time. I once worked on a system that wanted to store 18 months of archive data. This business touted this as a feature of our platform. No customer talked to needed more than 30 days. But I designed for 18 months, fast queries, etc because the business guys couldn’t define a real “Y”.

Once “Y” was set at 30 days, scaling it to 60 days or 90 days was trivial because the 30 day solution was so much more simply.

With that said, I also got the business guys to give me -
1.) a valid failure rate for archive data (basically on any given day given a normal flow of traffic, how much was I allowed to lose)
2.) an amount of time before the archive data had to be searchable (i.e immediately, 2 hours, a day?).
3.) What is the latency on searching? Does it have to be as fast as the real system? Or can it take some time?

With those 3 pieces of information, in my case I was able to offload archiving and searching onto a messaging system, and I only saved the delta. I implemented VCDIFF (http://www.faqs.org/rfcs/rfc3284.html) on the textual representation. I didn’t do anything fancy with the deltas. Just kept them in a chain (1 -> 2 -> 3 -> etc). Had Eric Sink’s article (http://www.ericsink.com/entries/time_space_tradeoffs.html) been around, I probably would have gone with a combination of key frame chains and flower chains.

Anyway – the deltas were stored in a BDB in a B-Tree database. I used a simple consistent hashing algorithm to pick the BDB to store to. Then I had a cron that ran every 1 hour to stop the message processing system, backup the BDB to a SAN, and restart the messaging system.

Search was also done through messaging. But my search was really easy. The client would put in their entity IDs, I would issue the query to the BDBs and aggregate the results. It typically took about 250 to 700 milliseconds (depending on network saturation).

For inserts we just record a single entry. For updates, we record the field name and old and new values. We use a stored procedure to create each change log entry. The stored procedure only records change log entries for fields with a changed value.

The set ID makes it easy to group the multiple change log entries that are created when multiple fields change in a single update.

We use a Python script to auto-generate the various triggers that are needed for each field in each table that is subject to change logging (there are some tables that don’t get any change logs and we don’t log changes to memo fields, for example).

uerying for the historical version of a particular table is very trivial and fairly easy. However you are correct in that in that is more expensive to query what a single user did. But it’s only N queries where N is the number of tables, which isn’t TOO bad for a single reporting-type query (you could cache or store the results).

A third approach could be to combine both ideas. The audit tables are a REALLY GOOD IDEA for doing this sort of archiving. However you could also store in a separate table all of the user operations which were performed and when. This way you can easily query what data a user touched and when. Then you can drill down into the historical data as needed.