Friday, July 25, 2008

Status update

We're awfully sorry for keeping the silence during last month. Here are some news:

We've started to refactor our Storage providers layer almost immediately after publishing the EAP. The published one was actually much more close to a test implementation, rather then to the real one, but it allowed us to make some tests working. What we have currently is much better, but it doesn't allow to run the tests yet. That's why our planned weekly nightly build updates aren't deliveded yet, although we hope to get new providers working relatively fast (2 weeks at max).

So what exactly we're doing:

1. Xtensive.Storage:

- The set of classes each providers may contain (i.e. a set of handlers) may vary now - i.e. we use only the ones we know, but we don't limit them explicitely. Some providers may contain additional handlers consumed by some services you can build. This approach also allows to implement "proxy providers" - i.e. a provider acting as another one, but exposing some additiona features.

- SessionHandler is changed dramatically. There are two basic methods now: Persist (persist a set of entities) and Select (returns a RecordSet by specified IndexInfo). So how entities are internally fetched now? By RSE queries (trees \ graphs of RecordSetProviders) - i.e. we build RSE queries fetching required entity fields right from the indexes (RecordSets provided by SessionHandler.Select method). Earlier we didn't implement this approach because it means we must be able to compile (better to say, transflrm) RSE queries into their equivalents which will execute in optimal way. E.g. in case with SQL storages we must transform such a query to a single SqlQueryProvider doing all the job on the server side; in case with in-memory storage we should do almost nothing.

- There are other severe refactorings in many other classes, but they are less important from the point of design & features.

2. Xtensive.Storage.Rse:

We significantly refactored it to introduce RSE query compilation. The primary case exposing its necessaty is described above.

Briefly, RSE query is almost absolutely the same as SQL query plan, but it is an extendable model. It looks strange we're going to "compile" the plan. In fact, we transform it from "stratighten-forward" (i.e. a plan that would be the most effecient if we could execute it locally) to "actual" (a plan that will be executed partially locally, and partially - distantly, i.e. on server(s)). Since not all the servers supports execution of our RSE plans (strange, yes? ;) actually I mean SQL servers), we should be able to do rather complex transforms during the compilation - in fact, compilation for SQL servers means we must be able to transform the whole graph of providers into a single one SqlQueryProvider instance doing all the underlying job.

The result we have now is the following:
- There is extendable set of "compilable" providers: they do nothing themselves, but ensure any request to them will be forwarded to their current compiled representation; "current" means you can have just a single instance of any RSE query during the application lifetime (i.e. store in some static variable) - it is compiled & cached in each Domain (yes, we finally renamed Storage back to Domain - Storage class in Xtensive.Storage namespace was violating the standard) in the optimal way.
- We have working compilers - "evaluating" (client-side) compiler is fully working, but compiler to SQL (i.e. to SqlDom) isn't fully working yet.
- Our compilation approach is designed to be quite extendable, and initially "understands" such aspects as client-side and server-side compilation. You can set up the hints (intermediate providers) in your RSE query specifying that source provider must be compiled in the way to be executed on the server, or on the client.
- New model allows to automatically pass the results between client and server using proxy providers (they're automatically "injected" into compiled graph where they're necessary).

So in fact, you now ask RSE to do something like this:
1) Join both result on server (implies that result of step 1.1 will be "exported" to the server)
1.1) Sort the result on client
1.1.1) Filter the result on client (where ...)
1.1.1.1) Select X.Value, X.ID from index X where (...) on server
1.2) Sort the result on server (where ...)
1.2.1) Select Y.Value, Y.ID from index Y where (...) on server

- Finally, we're keeping in mind that in future we must extend this to multiple servers.

3. Xtensive.Indexing:

We're slowly update this part - all the updates we do here aren't vital right now, but after the release they'll be ayway necessary. What's done:
- CompositeIndex seems to be fully implemented now
- We're improving Index.Serialize implementation - earlier it was fully implemented by a provider, but now it's core logic is implemented in the Index itself, and provider must just support it.

4. Xtensive.Integrity:

- We implement generic aspect based property-level contraints here. Type-level contraints were implemented earlier.

5. Xtensive.Core:

- A set of low-level aspects are generalized and moved from Storage.Aspects to Core.Aspects (ImplementProtectedConstructorAccessorAspect, ImplementConstrucorEpilogueAspect, ImlementAutoPropertyReplacementAspect)
- Some new useful abstractions are added, e.g. IHasExtensions, ExtensionCollection, IHasServices, Cached, etc.
- Some refactorings related to more frequent usage of extension methods.

6. Other:

- We started to write DataObjects.Net 4.0 Wiki. It will be available publicly before release. It's goes a bit further then a regular product Wiki - in particular, we describe all our standards there. It should allow anyone to not just use the DO, but to modify its source code.
- Some new decisions related to marketing of v4.0 are made. I'll announce them on this weekend.
- We've launched www.mescontrol.ru web site - it's actually completely in Russian and intended just for Russian audience, so we'd be glad to get the feedback on it from our Russian customers.
- Our team has grew up further - there are two more new persons (young, but promising).


Finally, we defined feature set we're going to deliver on release.

High priority:
- Support for 3 mapping strategies (mainly works now)
- Xtensive.Indexing: non-transactional in-memory indexes
- Compilation of RSE queries to Indexing (i.e. for client-side execution)
- Compilation of RSE queries to SQL
- Transactions
- Basic Undo-Redo (most likely - just to roll back the nested transactions for now)

Low priority:
- Basic versions of Entities (i.e. basic support for VersionID-like properties)
- EntitySet
- PairTo
- Our own serialization layer
- Storage-to-Storage syncronization
- Schema upgrade (most likely we'll initially support only create-remove actions )
- Custom property & type level constraints (validation)
- Basic support for Measures in RSE (Count & Size measures)

Post-release updates:
- Indexing: transactional indexes, file system indexes
- Full support for Measures in RSE
- LINQ to RSE compilation
- LINQ to SQL compilation
- Additional LINQ extensions


That's all for today. The most current goal for us is the next EAP update (2 weeks at max by plan).

No comments:

Post a Comment