Introduction

This tutorial demonstrates how to use the BaseConnector to host your own data inside Jangle. In particular, it shows how to export data from your ILSs borrower/patron table and import it into the BaseConnector.

The system used for this tutorial is Koha, an open source ILS. Koha was used as it is readily available for download, and makes a good reference implementation for experimenting with Jangle features.

Objectives

By the end of this tutorial, you will have:

1. Installed Jangle core
2. Added the BaseConnector to your Jangle installation
3. Exported borrower data from your ILS
4. Imported borrower data into the BaseConnector inside Jangle
5. Experimented with invoking the read-only services of the BaseConnector to display your borrower data

See DocumentationConventions for details of how the code and styles in this document should be interpreted.

Notes on the Base connector database

The BaseConnector adds tables to your Rails application's database tables, and also provides write access to those tables. However, it DOES NOT synchronise any changes you make back to the original source of the data. If you are following this tutorial to export your data and import it into a BaseConnector, you are best to treat Jangle as a source of read-only data.

If you want to use Jangle to both read and write your own data, you will need to use a read/write connector, rather than "borrowing" the BaseConnector's database.

Tutorial

Install Jangle core

You will need to install Ruby, Rails, Rails Engines and Jangle core. Please see the InstallationGuide for full details of how to do this.

Install the Base connector into your Jangle core

See these instructions.

Export data from your ILS

ILS systems usually provide some kind of backup script for their data; failing this, they will typically give you access to the back-end database itself, from which you can export your data. Ruby has good support for files containing data formatted using tab separation, so this would be a good format for your exported data.

You can write your export script using whichever programming language you like, or even use a tool to get your data into the right format. If you're using MySQL (the default database server for Koha), the simplest approach is to use a SELECT * FROM table INTO OUTFILE '/path/to/file' query, e.g. to dump Koha's borrower table you could do:

$ mysql -uroot -p
Enter password: ******
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 104
Server version: 5.0.38-Ubuntu_0ubuntu1.1-log Ubuntu 7.04 distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use koha;

Database changed
mysql> SELECT * FROM borrowers INTO OUTFILE '/tmp/borrowers.tsv';
Query OK, 5 rows affected (0.00 sec)

This dumps the data from the table in tab-separated format into the file /tmp/borrowers.tsv.

Import data into the Base connector

To get your data into the BaseConnector, you'll need to map your exported data onto the BaseConnector's schema for borrowers and addresses. Here's a graphical representation of that target schema:

Note that:

1. Each of these models maps onto a table in the database (model => table):
1. Actor => actors
2. Address => addresses
3. ActorAddress => actor_address
2. Each model has an id field which is not shown.
3. The ActorAddress model has two more fields, actor_id and address_id, which are not shown. These specify associations to the obvious models.
4. The Base connector uses a separate table for addresses: this enables different models to store addresses in the same table, and also potentially reuse them.
5. An actor is assigned an address for a period of time by creating a relationship between the actor and the address via the actor_address table. This enables keeping a historical record of someone's addresses, as well as storing as many addresses as you like for any individual.

You have two options for importing your data into the tables above:

1. Write your own SQL scripts to transfer your data between the databases. This isn't recommended, as it ties your import script to the BaseConnector's current database.
2. Use a Rails/ActiveRecord script to import the data from your CSV file. This is the preferred approach, and the one covered in the next section. The advantage of this is that it is database-agnostic, so you could point your Rails application at any ActiveRecord-compatible database and it would still work; and you don't need to write any SQL. Always a good thing.

(By the way, the graphic above was generated using Railroad to produce the dot file; then Graphviz to generate the png file. Like this:

$ railroad -o jangle_models.dot -M
$ dot -Tpng jangle_models.dot > jangle_models.png

Thanks to those projects.)

Using ActiveRecord to add data to the Base connector

Jangle's BaseConnector comes with a set of ActiveRecord model classes for representing a generic library domain. Your script can make use of these classes to import data without having to write SQL code; ActiveRecord will also validate any new records you add, and help you create associations between data in different tables.

You can experiment with this functionality from the command line using the console script:

$ ruby script\console
Loading development environment.
>> a = Actor.new        # Create a new actor and assign to the variable `a`
=> #<Actor:0xb6ff54b8 @new_record=true, @attributes={"updated_at"=>nil, "honorific"=>nil, "sex"=>nil, "family_name"=>nil, "given_name"=>nil, "other_names"=>nil, "telephone"=>nil, "date_of_birth"=>nil, "created_at"=>nil, "mobile_phone"=>nil, "email_address"=>nil}
>> a.given_name = "Elliot"         # Set given_name
=> "Elliot"
>> a.family_name = "Smith"         # Set family_name
=> "Smith"
>> a.date_of_birth = "12/12/1980"  # Set date_of_birth (not my real one :)
=> "12/12/1980"
>> a.save                          # Attempt to save
=> false                           # Validation error occurred
>> a.errors
=> #<ActiveRecord::Errors:0xb6fdf03c @base=#<Actor:0xb6ff54b8 @new_record=true, @attributes={"updated_at"=>nil, "honorific"=>nil, "sex"=>nil, "family_name"=>"Smith", "given_name"=>"Elliot", "other_names"=>nil, "telephone"=>nil, "date_of_birth"=>"12/12/1980", "created_at"=>nil, "mobile_phone"=>nil, "email_address"=>nil}, errors#<ActiveRecord::Errors:0xb6fdf03c ..., @errors={"sex"=>["is not included in the list"]}        # Invalid sex specified
>> a.sex = "M"
=> "M"
>> a.save
=> true                            # Saved successfully

Note that we're just using the field name followed by = to set a property on the object we created. When we call the save method on that object, Ruby saves it to the database, creating the SQL for us in the background.

The same approach can be used to create an address, e.g.

>> addr = Address.new({:street => "13 Unlucky Avenue", :locality => "Nowheresville"})
=> #<Address:0xb6f471d8 ...
>> addr.save
=> true

Then the address can be associated with the actor by creating a new ActorAddress which references the two objects we created above:

>> actor_address = ActorAddress.new({:actor => a, :address => addr, :active_from => Time.now})
=> #<ActorAddress:0xb6f26834 ...
>> actor_address.save
=> true

Scripting ActiveRecord to import data

Given how simple it is to write database code with ActiveRecord, it's relatively easy to put together a script to import data into any database which has a set of ActiveRecord model classes.

Here's an example which takes a TSV dump from the Koha borrowers table and imports it into the Jangle BaseConnector's actors, addresses and actor_address tables, using ActiveRecord models to do the heavy lifting:

class KohaBorrowersParser
  # The keys of this hash are Jangle Base field names;
  # the values are corresponding field positions in Koha's borrowers table
  @@jangle_base_actor_fieldnames = {
    'id' => 0,
    'given_name' => 3,
    'other_names' => 5,
    'family_name' => 2,
    'date_of_birth' => 18,
    'sex' => 33,
    'email_address' => 11,
    'telephone' => 10
  }
  
  @@jangle_base_address_fieldnames = {
    'street' => 7,
    'locality' => 9,
    'postal_code' => 45
  }
  
  # Read data a line at a time, adding a new actor, a new address, and
  # a record associating them for each
  def self.parse(data)
    data.each do |line|
      import_data = line.chomp.split(/\t/)
      
      # Create the actor
      actor = Actor.new
      @@jangle_base_actor_fieldnames.each do |field, index|
        value = import_data[index]
        value = nil if value.eql?('\N')
        actor.send((field + "=").to_sym, value)
      end
      save_object(actor)
      
      # Create their address
      address = Address.new
      @@jangle_base_address_fieldnames.each do |field, index|
        value = import_data[index]
        value = nil if value.eql?('\N')
        address.send((field + "=").to_sym, value)
      end
      save_object(address)
      
      # Associate them with their address
      actor_address = ActorAddress.new
      actor_address.actor = actor
      actor_address.address = address
      actor_address.active_from = Time.now
      save_object(actor_address)
    end
  end
  
  # Save an object and return its ID
  def self.save_object(obj)
    result = obj.save
    model = obj.class.name
    if result
      puts "#{model} added successfully} (id = #{obj.id})"
    else
      puts "Problems saving #{model}:"
      obj.errors.each { |field, msg| puts "\t#{field}: #{msg}" }
    end
    obj.id
  end

end

# The name of the TSV file containing borrowers exported from Koha
filename = "vendor/plugins/jangle_base_connector/doc/base_connector_with_local_data_tutorial/borrowers.tsv"

# Read the file, parse it, and add rows to the BaseConnector database
File.open(filename) do |f|
  KohaBorrowersParser.parse(f)
end

The code is available inside the Jangle BaseConnector plugin (vendor/plugins/jangle_base_connector/doc/base_connector_with_local_data_tutorial/koha_borrowers_importer.rb). Note that this is not intended as a generic, production-ready script with full error correction, but as a simple example of how to script against Jangle.

To run the import script, you'll need to have the Rails environment loaded. Rails provides a script for this called runner which loads up the correct environment and runs a script within that environment. For example, to run your own script, you would do the following (from the root of your Rails application):

$ ruby script\runner <script>.rb

View your borrower data through Jangle's web services

Once you have your data inside the Jangle BaseConnector, run the appropriate server start script (script\server or script\jangle) to start it. Then browse to http://localhost:3000/base/actors to see a list of your borrowers.