AsciiDatabase
Class AsciiTable

java.lang.Object
  |
  +--AsciiDatabase.AsciiTable

All Implemented Interfaces:

DatabaseTable, DatabaseTableBase

public final class AsciiTable

extends Object

implements DatabaseTable

A DatabaseTable that can represent any flat ascii file as a table for ModSQL to query. A table definition file describes the format of the ascii file.

Author:

chris.studholme@utoronto.ca

Field Summary

private static int	binfileVersion Version number of compiled table definition file.
private ColumnProperties[]	columns Properties of the table's columns.
private static String	compiledExtension Filename extension applied to compiled table definition files.
private int	dataoffset Offset of start of data within data files.
private AsciiReader[]	datastream List of data file to read data from.
private String	delimiter Delimiter for delimited files.
private int	filenum File where current pointer is located.
private int	maxdelimiter Max number of delimited fields.
private String	name Name of this table.
private int	ncolumns Number of columns.
private int	nfiles Total number of data files.
private int	nsourcerows Number of rows per data block.
private RowOption[][]	options Each row within a block can have multiple options applied to it before column data is extracted from it.
private byte[]	record Buffer for holding blocks of data.
private int	recordsize Size of fixed length records or -1 for variable length records.
private int	recordvalid Length of valid data within record.
private long	rowoffset Offset of current record within current file.
private int	sourceid Current row within the current record.
private SourceProperties[][]	sources Properties of the table's source rows.
private static int	startrecordsize Initial record size for variable length record files.
private long	tablesignature This table's signature.

Constructor Summary

AsciiTable(String tablename, String deffilename)
Open existing table.

AsciiTable(String tablename, String deffilename, String datafilename)
Opens existing table.

Method Summary

boolean	absolute(Object id) Seek to an absolute row id.
private static Number	Add(Number n, double d) Add a double to a Number.
void	addColumn(String name, int type, int maxlen) Add a column to the database.
void	AddDataFile(String datafilename) Add a new data file to the table.
void	addRow() Add a new row.
void	afterLast() Position pointer after last row.
void	beforeFirst() Position pointer before first row.
protected long	calculateTableSignature() Calculate a unique signature for this table.
void	close() Close the table.
void	commitUpdates() ModSQL will always call this method after performing a series of updateObject() calls on a particular row.
void	deleteRow() Delete the current row.
private void	delimitFields() Delimit fields.
private static Number	Divide(Number n, double d) Divide a Number by a double.
private static String	FilterForNumber(String s) Filter string of a number.
protected void	finalize() Close files.
int	findColumn(String name) Find a column by name.
boolean	findNext(int columnIndex, Object data) Find next row containing the specified data.
int	getColumnCount() Get count of columns.
int	getColumnDisplaySize(int column) Get ideal display width for column.
String	getColumnLabel(int column) Get an appropriate lable for the column.
String	getColumnName(int column) Get name of column.
int	getColumnType(int column) Get SQL type of column.
Object	getObject(int column) Read column data.
long	getRowCount() This method is expected to return the total number of rows in the database.
Object	getRowId() Get the id of the current row.
String	getTableName() Get table name.
long	getTableSignature() Get signature for table.
long	getTableSize() Get size of table.
boolean	isAfterLast() Is after last row?
boolean	isBeforeFirst() Is before first row?
boolean	isIndexAvailable(int column) Is a particular column indexed? No.
boolean	isReadOnly() Is the table read only? Yes.
private static Number	Multiply(Number n, double d) Multiply a double to a Number.
boolean	next() Advance to next row.
DatabaseIndex	openIndex(int column) Open index of specified column.
private void	ParseColumnDef(ColumnProperties column, SourceProperties source, String columndef, int offset) Parse an entire column definition.
private ColumnOption[]	ParseColumnOptions(String optionstring) Parse column options.
private RowOption	ParseRowOption(String optionstring) Parse a row option.
private int	ParseType(String type) Parse column type string.
private boolean	ReadCompiledFile(File filename) Read the compiled (binary) version of the table definition file.
private void	ReadRawTableDefinition(File filename) Read a raw (uncompressed) table definition file.
protected void	ReadTableDefinition(String filename) Read the table definition file.
private void	ResetTable(int file) Reset the table to before the beginning of the specified data file.
void	updateObject(int columnIndex, Object x) Change column data.
private void	WriteCompiledFile(File filename) Attemp to write the compiled (binary) version of the table definition file.
private static Number	Y2K(Number n, double d) Convert 2-digit year to 4-digit year.

Methods inherited from class java.lang.Object

, clone, equals, getClass, hashCode, notify, notifyAll, registerNatives, toString, wait, wait, wait

Field Detail

binfileVersion

private static final int binfileVersion

Version number of compiled table definition file.

compiledExtension

private static final String compiledExtension

Filename extension applied to compiled table definition files.

startrecordsize

private static final int startrecordsize

Initial record size for variable length record files. This record size is automatically double every time it is exceeded. It should reach its required length after the first few blocks are read from the data file.

datastream

private transient AsciiReader[] datastream

List of data file to read data from.

nfiles

private transient int nfiles

Total number of data files.

filenum

private transient int filenum

File where current pointer is located.

recordsize

private int recordsize

Size of fixed length records or -1 for variable length records.

delimiter

private String delimiter

Delimiter for delimited files.

maxdelimiter

private transient int maxdelimiter

Max number of delimited fields.

dataoffset

private int dataoffset

Offset of start of data within data files.

columns

private ColumnProperties[] columns

Properties of the table's columns.

ncolumns

private int ncolumns

Number of columns.

sources

private SourceProperties[][] sources

Properties of the table's source rows. Each block of data read from the data files can contain multiple rows of data, each of which can contain multiple columns.

nsourcerows

private int nsourcerows

Number of rows per data block.

options

private RowOption[][] options

Each row within a block can have multiple options applied to it before column data is extracted from it.

record

private transient byte[] record

Buffer for holding blocks of data.

recordvalid

private transient int recordvalid

Length of valid data within record.

rowoffset

private transient long rowoffset

Offset of current record within current file.

sourceid

private transient int sourceid

Current row within the current record.

name

private transient String name

Name of this table.

tablesignature

private transient long tablesignature

This table's signature. -1 indicates that the signature has not yet been calculated.

Constructor Detail

AsciiTable

public AsciiTable(String tablename,
                  String deffilename)
           throws IOException,
                  DatabaseException

Open existing table.

Parameters:

tablename - name of table

deffilename - path to table definition file

Throws:

IOException - if error occurs reading table definition

DatabaseException - if other error occurs

AsciiTable

public AsciiTable(String tablename,
                  String deffilename,
                  String datafilename)
           throws IOException,
                  DatabaseException

Opens existing table.

Parameters:

tablename - name of table

deffilename - path to table definition file

datafilename - path to first data file

Throws:

IOException - if error reading files

DatabaseException - if other error occurs

Method Detail

finalize

protected void finalize()

Close files. Just calls close().

Overrides:

finalize in class Object

AddDataFile

public void AddDataFile(String datafilename)
                 throws IOException,
                        DatabaseException

Add a new data file to the table. Data files ending with '.gz' are assumed to be GZIP compressed files.

Parameters:

datafilename - path to data file

Throws:

IOException - if error reading data file

DatabaseException - if other error occurs

ReadTableDefinition

protected void ReadTableDefinition(String filename)
                            throws DatabaseException,
                                   IOException

Read the table definition file. First an attempt is made to read the compiled version of the file. If this file does not exist, is unreadable, or is out of date, the raw table definition file is read instead. A new compiled version will be created (if possible).

Parameters:

filename - path to raw (uncompiled) table definition file

Throws:

IOException - if error reading file occurs

DatabaseException - if other error occurs

ReadCompiledFile

private boolean ReadCompiledFile(File filename)

Read the compiled (binary) version of the table definition file.

Parameters:

filename - File object pointing to binary file

Returns:

true if the file was successfully read, false otherwise

WriteCompiledFile

private void WriteCompiledFile(File filename)

Attemp to write the compiled (binary) version of the table definition file. No indication of failure is provided.

Parameters:

filename - File object point to binary file

ParseType

private int ParseType(String type)

Parse column type string. This test is case insensitive. Types supported include: char, int, long, single, double, bool.

Parameters:

type - string to parse

Returns:

value from java.sql.Types

ParseRowOption

private RowOption ParseRowOption(String optionstring)
                          throws DatabaseException

Parse a row option.

Parameters:

optionstring - String to parse

Returns:

RowOption object

Throws:

DatabaseException - if an error occurs

ParseColumnOptions

private ColumnOption[] ParseColumnOptions(String optionstring)
                                   throws DatabaseException

Parse column options. If optionstring starts with '#', it is ignored.

Parameters:

optionstring - String to parse

Returns:

ColumnOption[] array

Throws:

DatabaseException - if error occurs

ParseColumnDef

private void ParseColumnDef(ColumnProperties column,
                            SourceProperties source,
                            String columndef,
                            int offset)
                     throws IOException,
                            DatabaseException

Parse an entire column definition. Column definition is a line of text with the following columns:

   source_type source_length source_offset column_type column_name options

Obviously, options are optional. Any whitespace can seperate the columns.

Parameters:

column - ColumnProperties object to set

source - SourceProperties object to set

columndef - ascii line to parse

offset - to be added to source_offset before storing in SourceProperties

Throws:

IOException - if an error occurs

DatabaseException - if an error occurs

ReadRawTableDefinition

private void ReadRawTableDefinition(File filename)
                             throws DatabaseException

Read a raw (uncompressed) table definition file.

Parameters:

filename - File object pointing to raw file

Throws:

DatabaseException - if an error occurs

openIndex

public DatabaseIndex openIndex(int column)

Open index of specified column. Indexes are not currently supported.

Specified by:

openIndex in interface DatabaseTable

Parameters:

column - index of column to find index for

Returns:

DatabaseTable object for index, or null if column is not indexed

close

public void close()

Close the table. Closes all data files.

Specified by:

close in interface DatabaseTableBase

Following copied from interface: ModSQL.DatabaseTableBase

Throws:

DatabaseException - if a database-access error occurs

isIndexAvailable

public boolean isIndexAvailable(int column)

Is a particular column indexed? No. Indexes aren't supported yet.

Specified by:

isIndexAvailable in interface DatabaseTable

Parameters:

column - index of column that should be indexed

Returns:

true if the column is indexed, false otherwise

isReadOnly

public boolean isReadOnly()

Is the table read only? Yes. Read/write tables aren't supported yet.

Specified by:

isReadOnly in interface DatabaseTable

Returns:

true if the table is read only, false otherwise

getTableName

public String getTableName()

Get table name. This is just the name passed to the constructor.

Specified by:

getTableName in interface DatabaseTable

Returns:

table name

getColumnCount

public int getColumnCount()

Get count of columns.

Specified by:

getColumnCount in interface DatabaseTable

Returns:

number of columns in table

getTableSize

public long getTableSize()

Get size of table. This size should not be considered accurate. For GZIP compressed data files, the size returned is the total size of the compressed files. It is not possible to determine the size of the uncompressed data.

Specified by:

getTableSize in interface DatabaseTable

Returns:

total size of data files

calculateTableSignature

protected long calculateTableSignature()

Calculate a unique signature for this table. This signature should depend on the following:

• int recordsize
• int dataoffset
• ColumnProperties columns[]
• SourceProperties sources[][]
• RowOption options[][]
• int ncolumns
• int nsourcerows
• lengths and order of data files

Any change in the table that would invalidate an index should result in a change of this signature.

Returns:

table signature

getTableSignature

public long getTableSignature()

Get signature for table. In very rare circumstances, table signature will be recalculated everytime this function is called (if the signature just happens to be -1), otherwise the signature is cached.

Specified by:

getTableSignature in interface DatabaseTable

Returns:

table signature

getRowCount

public long getRowCount()

Description copied from interface: DatabaseTableBase

This method is expected to return the total number of rows in the database. This value is used by ModSQL for query optimization purposes. If the count cannot be easily determined, the database should return -1 to indicate that the count is unknown.

Specified by:

getRowCount in interface DatabaseTableBase

Following copied from interface: ModSQL.DatabaseTableBase

Returns:

number of rows in table, or -1 for unknown

Throws:

DatabaseException - if a database-access error occurs

findColumn

public int findColumn(String name)

Find a column by name.

Specified by:

findColumn in interface DatabaseTable

Parameters:

name - name of column to search for

Returns:

index of column, -1 if the column was not found

getColumnDisplaySize

public int getColumnDisplaySize(int column)

Get ideal display width for column. Only applied to VARCHAR columns.

Specified by:

getColumnDisplaySize in interface DatabaseTable

Parameters:

column - index of column

Returns:

width in characters

getColumnLabel

public String getColumnLabel(int column)

Get an appropriate lable for the column.

Specified by:

getColumnLabel in interface DatabaseTable

Parameters:

column - index of column

Returns:

label to be used for column

getColumnName

public String getColumnName(int column)

Get name of column.

Specified by:

getColumnName in interface DatabaseTable

Parameters:

column - index of column

Returns:

name of column

getColumnType

public int getColumnType(int column)

Get SQL type of column.

Specified by:

getColumnType in interface DatabaseTable

Parameters:

column - index of column

Returns:

value from java.sql.Types

addColumn

public void addColumn(String name,
                      int type,
                      int maxlen)
               throws DatabaseException

Add a column to the database. Not currently supported.

Specified by:

addColumn in interface DatabaseTable

Parameters:

name - name of new column

type - SQL type of new column

Throws:

DatabaseException - in case of error

delimitFields

private void delimitFields()

Delimit fields.

next

public boolean next()
             throws DatabaseException

Advance to next row.

Specified by:

next in interface DatabaseTableBase

Returns:

true if next row is valid, false otherwise

Throws:

DatabaseException - if an error occurs

findNext

public boolean findNext(int columnIndex,
                        Object data)
                 throws DatabaseException

Find next row containing the specified data.

Parameters:

columnIndex - index of column to contain data

data - data to search for

Returns:

true if the desired row was found, false otherwise

Throws:

DatabaseException - if an error occured

deleteRow

public void deleteRow()
               throws DatabaseException

Delete the current row.

Specified by:

deleteRow in interface DatabaseTableBase

Throws:

DatabaseException - if an error occured

addRow

public void addRow()
            throws DatabaseException

Add a new row.

Specified by:

addRow in interface DatabaseTable

Throws:

DatabaseException - if an error occured.

isBeforeFirst

public boolean isBeforeFirst()

Is before first row?

Specified by:

isBeforeFirst in interface DatabaseTableBase

Returns:

true if pointer is before the first row, false otherwise

isAfterLast

public boolean isAfterLast()

Is after last row?

Specified by:

isAfterLast in interface DatabaseTableBase

Returns:

true if pointer is after the last row, false otherwise

ResetTable

private void ResetTable(int file)
                 throws DatabaseException

Reset the table to before the beginning of the specified data file.

Note that this does not mean that if you reset to before the beggining of file 2, that the pointer is actually at the end of file 1.

Parameters:

file - to position pointer before

Throws:

DatabaseException - if an error occurs

beforeFirst

public void beforeFirst()
                 throws DatabaseException

Position pointer before first row.

Specified by:

beforeFirst in interface DatabaseTableBase

Throws:

DatabaseException - if error occurs

afterLast

public void afterLast()
               throws DatabaseException

Position pointer after last row.

Specified by:

afterLast in interface DatabaseTableBase

Throws:

DatabaseException - if error occurs

getRowId

public Object getRowId()

Get the id of the current row.

The row id is of the form [offset][file][source] where offset is most significant and source is least significant. It is not valid to compare row id's to determine if one row is before or after another. You can compare row id's to determine if two rows are the same row.

Specified by:

getRowId in interface DatabaseTableBase

Returns:

row id

absolute

public boolean absolute(Object id)
                 throws DatabaseException

Seek to an absolute row id. This method only accepts row id's as returned from getRowId().

Specified by:

absolute in interface DatabaseTable

Parameters:

id - row id to seek to

Returns:

true if the row was found, false otherwise

Throws:

DatabaseException - if an error occurs

Add

private static Number Add(Number n,
                          double d)
                   throws DatabaseException

Add a double to a Number. The result is an object of the same type as n.

Parameters:

n - Number to add

d - double to add to the Number

Returns:

Number object of same class as n

Throws:

DatabaseException - if n is an invalid type

Multiply

private static Number Multiply(Number n,
                               double d)
                        throws DatabaseException

Multiply a double to a Number. The result is an object of the same class as n.

Parameters:

n - Number to multiply

d - double to multiply to the Number

Returns:

Number object of same class as n

Throws:

DatabaseException - if n is an invalid type

Divide

private static Number Divide(Number n,
                             double d)
                      throws DatabaseException

Divide a Number by a double. The result is an object of the same class as n.

Parameters:

n - Number to divide

d - double to divide Number by

Returns:

Number object of same class as n

Throws:

DatabaseException - if n is an invalid type

Y2K

private static Number Y2K(Number n,
                          double d)
                   throws DatabaseException

Convert 2-digit year to 4-digit year.

Parameters:

n - date to adjust

d - maximum date to consider as 2000+date

Returns:

Integer object representing new date

Throws:

DatabaseException - if n is an invalid type

FilterForNumber

private static String FilterForNumber(String s)
                               throws DatabaseException

Filter string of a number. This method does two things:

1. filter out non-numeric characters
2. if a + or - is on the end of the string, move it to the front

Note: 2 is a hack to solve problems with our ROB database

Parameters:

s - string to filter

Returns:

filtered string

Throws:

DatabaseException - if an error occurs

getObject

public Object getObject(int column)
                 throws DatabaseException

Read column data.

Specified by:

getObject in interface DatabaseTableBase

Parameters:

column - index of column to read

Returns:

Object representing column data

Throws:

DatabaseException - if an error occurs

updateObject

public void updateObject(int columnIndex,
                         Object x)
                  throws DatabaseException

Change column data.

Specified by:

updateObject in interface DatabaseTableBase

Parameters:

columnIndex - index of column to change

x - Object to set column to

Throws:

DatabaseException - if an error occurs

commitUpdates

public void commitUpdates()
                   throws DatabaseException

Description copied from interface: DatabaseTableBase

ModSQL will always call this method after performing a series of updateObject() calls on a particular row. If this call is not received before the row is changed or the table is closed, the changes made by updateObject() should be discarded. In the case where addRow() was called before the updateObject() calls, the new row can be discarded as well.

Specified by:

commitUpdates in interface DatabaseTableBase

Following copied from interface: ModSQL.DatabaseTableBase

Throws:

DatabaseException - if a database-access error occurs