Sunday, July 27, 2008

DataObjects.NET v4.0 will be available under GPL

That's one of the most important decisions we've made. This does not mean we'll eliminate the existing licenses - we'll keep them, so anyone will be able to chose between "non-viral" paid or "viral" free licenses.

By doing this we want to reach several goals:
- Enter the market faster - we already spend much more time on its development then we planned, so we want to take it back here.
- Make the product much more popular. If we'd compare this to something existing, which is closer to us by the underlying ideas and approach, we want to get nearly the same effect as for Ruby on Rails.
- Build bigger community around the product. In particular, we'll allow other people to contribute to the product, as well as to its Wiki.
- Make it stable earlier. Source code should allow the people to help us debugging the issues.
- Show it is good inside, not just outside - i.e. there is nothing "dark" we prefer to hide. We'll publish it with complete set of tests. The quality of its code is really much better then of v3.9.
- Get it developed further faster - as we wrote, it will initially go out without some features (I mean mainly LINQ) that are already implemented in many of its competitors, although as a platform itself it looks as the most advanced one (none of competitors neither have full-featured built-in database engine, nor plans for supporting not just N-tier, but a distributed environment). And we hope it will help us to get them implemented faster.
- Make the product more competitive - here I mean mainly other free products, such as EF. We understand it will be hard to compete with it, but on the other hand, we're creating a new niche one the market: DataObjects.Net will be the only ORM platform that needs no database at all (sounds a bit funny, but on the other hand, anyone using ORM dreams to forget about the database, and we really make this dream come true now ;) ), and works without any lacks in such a configuration. I suspect that e.g. EF will never support this - I don't believe they will compete with SQL Server. Although in wide range of cases (e.g. in Silverlight applications) this is quite desirable.

Availability under GPL doesn't mean we're going to get rid of this product and of its further development, as it frequently happens. On contrary, we're doing this to ensure it will have a good future. We are releasing an new innovative technology having much safer position as a company now then 5 years ago (actually we were much closer to the group of yound people that time, then to a company) - and this time we are sure this is the way we should and can choose.

It doesn't an "all or nothing" game for us - we're already recognized as good software developers at least, and have enough potential to grow further even without DataObjects.Net. But for the new version of DataObjects.Net it really is - i.e. such an approach should expose its full potential faster, and we believe it will be a good race for it. It should swim up and seriously affect on the market or sink fast (6 months to 1 year is enough time to see this) - and personally I don't believe it can sink.

So let's see how it swims up and gets its positions back!

Friday, July 25, 2008

Status update

We're awfully sorry for keeping the silence during last month. Here are some news:

We've started to refactor our Storage providers layer almost immediately after publishing the EAP. The published one was actually much more close to a test implementation, rather then to the real one, but it allowed us to make some tests working. What we have currently is much better, but it doesn't allow to run the tests yet. That's why our planned weekly nightly build updates aren't deliveded yet, although we hope to get new providers working relatively fast (2 weeks at max).

So what exactly we're doing:

1. Xtensive.Storage:

- The set of classes each providers may contain (i.e. a set of handlers) may vary now - i.e. we use only the ones we know, but we don't limit them explicitely. Some providers may contain additional handlers consumed by some services you can build. This approach also allows to implement "proxy providers" - i.e. a provider acting as another one, but exposing some additiona features.

- SessionHandler is changed dramatically. There are two basic methods now: Persist (persist a set of entities) and Select (returns a RecordSet by specified IndexInfo). So how entities are internally fetched now? By RSE queries (trees \ graphs of RecordSetProviders) - i.e. we build RSE queries fetching required entity fields right from the indexes (RecordSets provided by SessionHandler.Select method). Earlier we didn't implement this approach because it means we must be able to compile (better to say, transflrm) RSE queries into their equivalents which will execute in optimal way. E.g. in case with SQL storages we must transform such a query to a single SqlQueryProvider doing all the job on the server side; in case with in-memory storage we should do almost nothing.

- There are other severe refactorings in many other classes, but they are less important from the point of design & features.

2. Xtensive.Storage.Rse:

We significantly refactored it to introduce RSE query compilation. The primary case exposing its necessaty is described above.

Briefly, RSE query is almost absolutely the same as SQL query plan, but it is an extendable model. It looks strange we're going to "compile" the plan. In fact, we transform it from "stratighten-forward" (i.e. a plan that would be the most effecient if we could execute it locally) to "actual" (a plan that will be executed partially locally, and partially - distantly, i.e. on server(s)). Since not all the servers supports execution of our RSE plans (strange, yes? ;) actually I mean SQL servers), we should be able to do rather complex transforms during the compilation - in fact, compilation for SQL servers means we must be able to transform the whole graph of providers into a single one SqlQueryProvider instance doing all the underlying job.

The result we have now is the following:
- There is extendable set of "compilable" providers: they do nothing themselves, but ensure any request to them will be forwarded to their current compiled representation; "current" means you can have just a single instance of any RSE query during the application lifetime (i.e. store in some static variable) - it is compiled & cached in each Domain (yes, we finally renamed Storage back to Domain - Storage class in Xtensive.Storage namespace was violating the standard) in the optimal way.
- We have working compilers - "evaluating" (client-side) compiler is fully working, but compiler to SQL (i.e. to SqlDom) isn't fully working yet.
- Our compilation approach is designed to be quite extendable, and initially "understands" such aspects as client-side and server-side compilation. You can set up the hints (intermediate providers) in your RSE query specifying that source provider must be compiled in the way to be executed on the server, or on the client.
- New model allows to automatically pass the results between client and server using proxy providers (they're automatically "injected" into compiled graph where they're necessary).

So in fact, you now ask RSE to do something like this:
1) Join both result on server (implies that result of step 1.1 will be "exported" to the server)
1.1) Sort the result on client
1.1.1) Filter the result on client (where ...)
1.1.1.1) Select X.Value, X.ID from index X where (...) on server
1.2) Sort the result on server (where ...)
1.2.1) Select Y.Value, Y.ID from index Y where (...) on server

- Finally, we're keeping in mind that in future we must extend this to multiple servers.

3. Xtensive.Indexing:

We're slowly update this part - all the updates we do here aren't vital right now, but after the release they'll be ayway necessary. What's done:
- CompositeIndex seems to be fully implemented now
- We're improving Index.Serialize implementation - earlier it was fully implemented by a provider, but now it's core logic is implemented in the Index itself, and provider must just support it.

4. Xtensive.Integrity:

- We implement generic aspect based property-level contraints here. Type-level contraints were implemented earlier.

5. Xtensive.Core:

- A set of low-level aspects are generalized and moved from Storage.Aspects to Core.Aspects (ImplementProtectedConstructorAccessorAspect, ImplementConstrucorEpilogueAspect, ImlementAutoPropertyReplacementAspect)
- Some new useful abstractions are added, e.g. IHasExtensions, ExtensionCollection, IHasServices, Cached, etc.
- Some refactorings related to more frequent usage of extension methods.

6. Other:

- We started to write DataObjects.Net 4.0 Wiki. It will be available publicly before release. It's goes a bit further then a regular product Wiki - in particular, we describe all our standards there. It should allow anyone to not just use the DO, but to modify its source code.
- Some new decisions related to marketing of v4.0 are made. I'll announce them on this weekend.
- We've launched www.mescontrol.ru web site - it's actually completely in Russian and intended just for Russian audience, so we'd be glad to get the feedback on it from our Russian customers.
- Our team has grew up further - there are two more new persons (young, but promising).


Finally, we defined feature set we're going to deliver on release.

High priority:
- Support for 3 mapping strategies (mainly works now)
- Xtensive.Indexing: non-transactional in-memory indexes
- Compilation of RSE queries to Indexing (i.e. for client-side execution)
- Compilation of RSE queries to SQL
- Transactions
- Basic Undo-Redo (most likely - just to roll back the nested transactions for now)

Low priority:
- Basic versions of Entities (i.e. basic support for VersionID-like properties)
- EntitySet
- PairTo
- Our own serialization layer
- Storage-to-Storage syncronization
- Schema upgrade (most likely we'll initially support only create-remove actions )
- Custom property & type level constraints (validation)
- Basic support for Measures in RSE (Count & Size measures)

Post-release updates:
- Indexing: transactional indexes, file system indexes
- Full support for Measures in RSE
- LINQ to RSE compilation
- LINQ to SQL compilation
- Additional LINQ extensions


That's all for today. The most current goal for us is the next EAP update (2 weeks at max by plan).