Archive for the ‘nhibernate’ tag
Saved by a yield – Bulking NHibernate Read-only Data
I’ve just finished working on a service which exports information from a legacy database once per day, in a nutshell retrieve, map, publish.
In a four-step process it works as follow:
- Query the data layer for records modified after last run date (~24hrs)
- Load the records into a domain model (responsibility of NHibernate)
- Convert the collection to our flat shared schema DTOs (AutoMapper)
- Publish the messages to the bus (NServiceBus / MSMQ)
Whilst performance wasn’t a noted priority it still had to complete the daily job within a few hours and with the resources a basic server afforded. In the initial stages of implementation, I had my integration tests running against a SQLite database and performance wise everything was progressing smoothly. However when we pointed it against a production database (SQL Server) with a six figure number of aggregate instead of less than 100 for test data we soon hit problems, dreaded out of memory exceptions.
It didn’t take long to pinpoint the issue was related the data access strategy, bringing all the entities out of the database at once made the Session balloon at shocking rate, in a hindsight a bad approach to begin with. This was a showstopper, and with it approaching 5pm, and happy hour on the beach finishing at 7pm, I could already see this turning out to be disappointing.
I contemplated a few ideas, changing some settings with regards to mutability (we are doing read-only), switching to an IStatelessSession, and even polluting the consumer to use some paging mechanism…. I played around for while and then I thought about a yield inside the service layer, essentially implementing my own iterator to keep the unit of work small and thus underlying NHibernate session minimal.
In my service layer and wherever possible I favour IEnumerable over IList, for the definition I had:
IEnumerable<Derivative> GetDerivativesWithPriceChanges(DateTime laterThan)
And for the code (refactor due):
public IEnumerable<Derivative> GetDerivativesWithPriceChanges(DateTime laterThan) { Expression<Func<Derivative, bool>> predicate = d => d.PricingUpdated > laterThan; int bufferSize = 10; // make configurable, inject int marker = 0; int lastAmountRetrieved = 0; do { using (UOW.Start()) { var selection = new Func<IQueryable<Derivative>, IQueryable<Derivative>> (d => d.Skip(marker).Take(bufferSize)); var derivatives = derivativeRepository.GetByQuery(predicate, selection).ToList(); lastAmountRetrieved = derivatives.Count; foreach (var derivative in derivatives) { yield return derivative; } } marker = marker + bufferSize; } while (lastAmountRetrieved == bufferSize); yield break; }
)
Inside GetDerivativesWithPriceChanges, I’m paging through the database opening and closing the Unit of work after a specified number of entities are returned. Note: UOW system provides the underlying ISession to the repositories, although my UOW differs slights I recommend analysing Ayende’s Rhino Commons for a good example.
Best of all the consumer of the service is none the wiser, the code for paging was already on the base repository (as simple as Skip(x).Take(y) with NHibernate.Linq), and as an added benefit we have quicker feedback to our logging system and on our integration tests. It’s certainly not the purist implementation, but since I was working with read-only data it turned out perfect for this scenario and I was enjoying a Heineken by the beach by 6.30pm. I imagine I will eventually refactor the some of the logic for readonly paging/uow streaming to its own class, but for today, check-in, and close solution. 
A few lessons learnt:
- Memory is
notnever infinite. - Long running operations are bad for integration tests, these tests are often already a magnitude slower than unit tests so keep them trim.
- Always consider the data and quantity of which you’ll be working with in production. If unsure, ask.
- The problem is rarely solely NHibernate but usually the high-level approach taken to tackle the problem, don’t damn the NHibernate Session.
NHibernate & Linq – Free yourself from NHibernate
Note: This is tested against 2.0 not the 2.1 Beta. Today I’ve been experimenting with NHibernate.Linq against the GA release, there are apparently a few issues and a new version under development which takes a different approach rather than parsing to the criteria API, but hopefully the public API should be the similar whatever version gets released.
In the past I’ve hated leaking ICriteria/DetachedCriteria out of the repository, in addition with a lot of the business logic being pushed down into infrastructure code it always felt a little dirty. Whatever approach I’ve taken there has always been some aspect that didn’t feel right, I’ve always been concerned that I’ve have NH—>DB rather than it just being an implementation of the infrastructure that’s wired in at the end, thankfully with NHibernate and Linq there is some light at the end of the tunnel. I’ve haven’t finalised my approach yet, I’ve still got some internal debating to do about location of the query and services and possibly the integrating the specification pattern (now it’s easy to consume), but I think I’m going to take a slight reversal on my previous post.
This is what I’m liking.
public virtual IEnumerable<T> GetByQuery (Expression<Func<T, bool>> predicate, Func<IQueryable<T>, IQueryable<T>> selectionPredicate) { using (ITransaction tx = Session.BeginTransaction()) { try { IQueryable<T> query = Session.Linq<T>().Where(predicate); return selectionPredicate(query); } catch (HibernateException hibernateException) { tx.Rollback(); throw new RepositoryException(hibernateException); } } }
In my service layer or outer repository etc..:
Expression<Func<Hotel, bool>> predicate = hotel=>
hotel.Stars > 3 && hotel.Country == Country.UK
Func<IEnumerable<Hotel>,IEnumerable<Hotel>> selectionPredicate = col =>
col.Skip(10).Take(10); // pagingSetup.GetQueryForPage(requestedPage);
var hotels = hotelRepository.GetByQuery(predicate, selectionPredicate);
return hotels;
I fired up Ayende’s cool NH Profiler and the above does get converted to a single neat parameterized SQL statement with correct selection of the paging. This also significantly aids in testability as injecting a in-memory repository in place of NH into the service layer is now super easy just execute the expressions as is on the collection (I won’t even post the code for that)… In future I might not even be working with NHibernate – so this approach gives me great flexibility with no query logic trapped in HQL or the NH criteria api – who knows – it could be couchdb etc..
NHibernate 2.1 Trunk – Entity-Name, Some Inheritance and Dynamic Component
I have been experimenting with the NHibernate trunk today to test the feasibility of a dynamic persistence scenario. To follow this blog post it’s probably a good idea to read my previous post about dynamic-component too and to have a version of the NHibernate trunk to play with. To get the latest trunk point your subversion client at https://nhibernate.svn.sourceforge.net/svnroot/nhibernate/trunk/nhibernate/ and follow the build instructions included.
Working with a Person entity again:
1:
2: namespace Blog.Example.Domain
3: {
4: public class Person
5: {
6: public virtual int Id { get; protected set; }
7:
8: public Person()
9: {
10: Details = new Dictionary<string, object>();
11: }
12:
13: public virtual string Name { get; set; }
14: public virtual string Country { get; set; }
15: public virtual IDictionary Details { get; protected set; }
16:
17:
18: }
19: }
I need this Person entity to persist across multiple tables, one base table called Person and then a separate table to store the information in the Details property, this separate table will be different depending on the Country property of the Person.
So a Person object representing “Tony Blair” with UK set a country would be persisted over the tables Person and PersonUK, and a Person object representing “George Bush” would be persisted over the tables Person and PersonUS, people from different countries will have different information stored in the Details property bags which will be persisted to explicit database columns (i.e. the database schema is not dynamic in nature). As an Nhibernate user you’re probably recognising a possible solution of using joined-subclass and creating two empty derived types called PersonUK and PersonUS and using a Country as the database level discriminator and obviously the Type as the discriminator in the object model. In a normal domain model this would suffice rather well, but in our requirements we need to define new types of Person without recompilation, essentially at runtime we might need to setup mappings for a Person entity to represent french people that persists across Person and Person_FR tables and at such point we do not have a PersonFrench class (True, we could probably create class for each country to somewhat future proof this scenario but my actual real-world version of this is a little more convoluted unfortunately).
This is where the current general availability release of NHibernate starts to show some weaknesses, but not to worry they have been address with the trunk and forthcoming in the 2.1 GA release. In the current release one can not map the same type to a different choice of tables, one will encounter DuplicateMappingException, as NHibernate works heavily around using the Type to identify which entity working its with, at this point the options dwlindle to forking NHibernate to handle types generated at runtime or generate and persist an assembly before firing up or replacing a SessionFactory, both are far from ideal and will probably throw further roadblocks later in the project. If it feels bad it probably is!
The latest trunk introduces a new mapping attribute the works on several class related mapping elements called entity-name that basically assigns as an identifier for each mapping strategy, this allows one to have, for lack of better terminology, different profiles for persisting the same entity type in different ways for example to different tables and and even with different choice of properties. We use entity-name on the different joined subclasses of Person rather than the CLR type of subclass, in our case we do not even use a different class in our object model, since our Person domain object can represent all our different styles of Person thanks to the property bag. We use the element-name to define different profiles mapping different properties to our dynamic-component based on the a country identifier.
This is our new mapping file, notice the inclusion of entity-name on joined-subclass:
1: <?xml version="1.0" encoding="utf-8" ?>
2: <hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
3: assembly="Blog.Example.Domain"
4: namespace="Blog.Example.Domain"
5: >
6: <class name="Person" table="Person" dynamic-update="true">
7: <id name="Id" type="Int32" column="PersonId">
8: <generator class="identity" />
9: </id>
10: <discriminator column="Country"/>
11:
12: <property name="Name" />
13: <property name="Country" column="Country" />
14:
15: <joined-subclass check="none" name="Person" entity-name="US" table="PersonUS">
16: <key column="PersonId" not-null="true"></key>
17: <dynamic-component name="Details">
18: <property name="NumberOfLawSuitsFilled" type="Int32"/>
19: <property name="SocialSecurityNumber" type="String"/>
20: </dynamic-component>
21: </joined-subclass>
22:
23: <joined-subclass check="none" name="Person" entity-name="UK" table="PersonUK">
24: <key column="PersonId" not-null="true"></key>
25: <dynamic-component name="Details">
26: <property name="DVLAReference" type="String"/>
27: <property name="NationalInsuranceNumber" type="String"/>
28: </dynamic-component>
29: </joined-subclass>
30:
31: </class>
32: </hibernate-mapping>
Using the mapping above the same Person entity we defined at the start of this blog can now represent multiple types of Person, persisted across different tables that isn’t depended on new derived types to satisfy NHibernate. To handle our dynamic requirement we use an interceptor to tell NHibernate what entity-name profile to use in the persistence process (the retrieval identifies which joined-subclass to use by the country database level discriminator).
1: using Blog.Example.Domain;
2: using NHibernate;
3:
4: namespace Blog.Example.Host
5: {
6: public class PersonInterceptor : EmptyInterceptor
7: {
8: public override string GetEntityName(object entity)
9: {
10: Person person = entity as Person;
11:
12: if (person != null)
13: {
14: return person.Country;
15: }
16:
17: return base.GetEntityName(entity);
18: }
19:
20: }
21: }
In our scenario we’ll amend the mapping file on the fly adding new joined-sublass/element-name parts when we’re notified of a new type of Person and the metadata associated with it (columns/properties/table-name), the above combination affords us this without creating new classes or amending our domain object model. We can even regenerate the sessionfactory on demand during our application lifecycle, allowing our database schema to evolve (should you be unfortunately be presented with such legacy scenarios), without the need for recompilation or even a restart.
In a nutshell this is a pretty cool addition and can be used in many different ways, in a more simple example one could create different profiles with different fetching strategies, e.g. lazy loading on relations for web app and all eagerly fetched on internal reporting etc.., in the same mapping assembly. The new version of NHibernate provides overloads on most of Session method to specify the entity-name profile in situations where you don’t want to use an Interceptor to determine this, this explicit way has limitations when it comes different entity-names on cascading saves and updates.
You can download the Visual Studio 2008 Solution I’ve put together with a working example to get more hands-on idea. You’ll need to run the blogexample.sql script in the artifacts solution folder (relative root folder on file system) against your local SQL Express instance, and/or edit the hibernate.cfg.xml accordingly. Enjoy.
Fluent-NHibernate Contribution
Expanding on my recent post of dynamic-component, I’ve committed a contribution to the open source Fluent NHibernate project which has just been integrated into the trunk. Thanks James.
NHibernate mapping with dynamic-component
One of the more obscure features of NHibernate 2.0 I’ve found, that is somewhat lacking in documentation and certainly in the fame it deserves, is dynamic-component, it has just a one-liner in the official docs and few sparse mentions elsewhere. In it’s simplest forms, thinking in terms of constructing your NHibernate mapping, when you define mapping elements inside a dynamic-component element it adds each of them to an IDictionary object rather than binding them to CLR properties, i.e. the name attribute refers to the key on the dictionary entry instead.
This provides one way for you to define entities that contain property bags mapped to explicit columns on the database, either in the same table or in a different table using your standard join or joined-subclass mapping elements.
Jumping straight into a primitive example, whilst this may be a little trivial and its applicability questionable, the real magic follows later when combined with other tools at your disposal.
1: using System.Collections;
2: using System.Collections.Generic;
3:
4: namespace Blog.Example.Domain
5: {
6: public class Person
7: {
8: public virtual int Id { get; protected set; }
9:
10: protected Person() {}
11:
12: public Person(string name)
13: {
14: Name = name;
15: Details = new Dictionary<string, object>();
16: }
17:
18: public virtual string Name { get; protected set; }
19: public virtual IDictionary Details { get; protected set; }
20:
21:
22: }
And define a mapping file, pretty standard conventions, but now including a dynamic-component:
1: <?xml version="1.0" encoding="utf-8" ?>
2: <hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
3: assembly="Blog.Example.Domain"
4: namespace="Blog.Example.Domain"
5: >
6: <class name="Person" table="Person"
7: dynamic-insert="true"
8: dynamic-update="true">
9: <id name="Id" type="Int32" column="PersonId">
10: <generator class="identity" />
11: </id>
12: <property name="Name" />
13: <dynamic-component name="Details">
14: <property name="SocialNumber" type="String"/>
15: <property name="DVLANumber" type="String"/>
16: </dynamic-component>
17: </class>
18: </hibernate-mapping>
And a test wrapper to ensure everything works (this isn’t a great example of test design, this is purely for illustration):
1: using System;
2: using System.Diagnostics;
3: using Blog.Example.Domain;
4: using NUnit.Framework;
5: using NUnit.Framework.SyntaxHelpers;
6:
7: namespace Blog.Example.Host.IntTests
8: {
9: [TestFixture]
10: public class PersonDatabaseTests
11: {
12:
13: [Test]
14: public void can_save_new_instance_of_person_and_retrieve()
15: {
16: Person person = new Person("Matt");
17: person.Details.Add("SocialNumber","A730399/0");
18: person.Details.Add("DVLANumber","100-153-200");
19:
20: var session = new SessionProvider().GetSession();
21: session.SaveOrUpdate(person);
22:
23: Assert.That(person.Id, Is.Not.EqualTo(0));
24:
25: int personId = person.Id;
26:
27: session.Clear();
28:
29: Person fetchedPerson = session.Load<Person>(personId);
30: Assert.That(fetchedPerson.Details.Contains("SocialNumber"),Is.True);
31: Assert.That(fetchedPerson.Details.Contains("DVLANumber"),Is.True);
32:
33: OutputPersonDetails(fetchedPerson);
34:
35: }
36:
37: static void OutputPersonDetails(Person subject)
38: {
39: foreach (string detailKey in subject.Details.Keys)
40: {
41: Trace.WriteLine(String.Format("{0} - {1}", detailKey, subject.Details[detailKey]));
42:
43: }
44: }
45:
46: }
47: }
As you can probably gather this inserts the values in the dictionary in correct columns as defined by the mapping file, and also verifies the retrieval too. Hypothetically, in cases where your domain object design must remain static providing an extension data property bag provides some interim flexibility between major releases, of course you lose some of type safety afforded by properties but when you are integrating at a database level (I don’t condone this) and other applications are evolving the database schema perhaps this could be quite handy in moving some of the extra new data through your application.
My real interest in the dynamic-property I’ll demonstrate with more depth in the next article when we look more a dynamic scenario and a less isolated example.
