Performance

CSV Manager is a data processing component so good scalable performance is an important design requirement. We use a randomised set of CSV files, each 10 times larger than the first, to test the performance of the component.

The performance test is run on a Celeron 1200 CPU, with the default java 1.4.2 runtime from Sun Microsystems. We obtained an average of over 1000 rows per second when using the parser alone. Additional row operations such as performing database INSERTS will of course reduce this figure significantly.

Test Files

A typical row in each test file looks like this:

E,169135334,8804674298342089685,0.7341090379151565, \
  Wed Mar 10 20:46:38 GMT 233962134,"K05z.l@ya^0nUyh_YqN]@`9);$:7S:h7u,RedF I"

Files of size 10 to 100000 rows are created, and the four standard input methods are used: List of Strings, List of Lists, ResultSet and TableModel. Each file is read 10 times and the timing results averaged.

Results

10 Rows, time in milliseconds

Where the bars, from left to right, correspond to:

•   Load method
•   LoadAsLists method
•   LoadTableModel method
•   LoadResultSet method

The remaining results are:

100 Rows, time in milliseconds

1000 Rows, time in milliseconds

10000 Rows, time in seconds

100000 Rows, time in seconds

Analysis

First we divide the time taken by the number of rows, to demonstrate that the row rate remains basically constant.

Second, we take log₁₀ of the times, to show that the parsing algorithm is 0(N).

Data

The original data used to produce these graphs is also available as a CSV file so that you can verify the results for yourself. In addition, we provide the source code of the performance test class and the test file creator class so that you can run the tests to confirm these results.