go: github.com/openmpp/go/dbcopy Files

Command dbcopy

dbcopy is command line tool for import-export OpenM++ model metadata, input parameters and run results.

Arguments for dbcopy can be specified on command line or through .ini file:

dbcopy -ini my.ini

Command line arguments take precedence over ini-file options.

Only model argument does not have default value and must be specified explicitly:

dbcopy -m modelOne
dbcopy -dbcopy.ModelName modelOne
dbcopy -dbcopy.ModelDigest 649f17f26d67c37b78dde94f79772445

Model digest is globally unique and you may want to it if there are multiple versions of the model.

There are 5 possible copy directions: "text", "db", "db2db", "csv", "csv-1". By default it is copy to "text". It is also possible to delete entire model or some model data from database (see dbcopy.Delete below).

Copy to "text": read from database and save into metadata .json and .csv values (parameters and output tables):

dbcopy -m modelOne

Copy to "db": read from metadata .json and .csv values and insert or update database:

dbcopy -m modelOne -dbcopy.To db

Copy to "db2db": direct copy between two databases:

dbcopy -m modelOne -dbcopy.To db2db -dbcopy.ToDatabase "Database=dst.sqlite;OpenMode=ReadWrite"

Copy to "csv": read entire model from database and save .csv files:

dbcopy -m modelOne -dbcopy.To csv

Separate sub-directory created for each input set and each model run results.

Copy to "csv-1": read entire model from database and save .csv files:

dbcopy -m modelOne -dbcopy.To csv-1

It dumps all input parameters sets into all_input_sets/parameterName.csv files. And for all model runs input parameters and output tables saved into all_model_runs/tableName.csv files.

By default entire model data is copied. It is also possible to copy only: model run results and input parameters, set of input parameters (workset), modeling task metadata and run history.

To copy only one set of input parameters:

dbcopy -m redModel -dbcopy.SetName Default
dbcopy -m redModel -s Default

To copy only one model run results and input parameters:

dbcopy -m modelOne -dbcopy.RunId 101
dbcopy -m modelOne -dbcopy.RunName MyFirstRun

To copy only one modeling task metadata and run history:

dbcopy -m modelOne -dbcopy.TaskId 1
dbcopy -m modelOne -dbcopy.TaskName taskOne

It may be convenient to pack (unpack) text files into .zip archive:

dbcopy -m modelOne -dbcopy.Zip=true
dbcopy -m modelOne -dbcopy.Zip
dbcopy -m redModel -dbcopy.SetName Default -dbcopy.Zip

By default model name is used to create output directory for text files or as input directory to import from. It may be a problem on Linux if current directory already contains executable "modelName".

To specify output or input directory for text files:

dbcopy -m modelOne -dbcopy.OutputDir one
dbcopy -m redModel -dbcopy.OutputDir red -s Default
dbcopy -m redModel -dbcopy.InputDir red -dbcopy.To db -dbcopy.ToDatabase "Database=dst.sqlite;OpenMode=ReadWrite"
dbcopy -m redModel -dbcopy.OutputDir red -s Default

To avoid path conflicts InputDir and OutputDir combined with model name, model run name or name of input parameters set. For example:

dbcopy -m redModel -dbcopy.OutputDir red -s Default

will place "Default" input set of parameters into directory red/redModel.set.Default. If neccesary you can specify exact directory for input parameters by using "-dbcopy.ParamDir" or "-p":

dbcopy -m modelOne -dbcopy.SetId 2 -dbcopy.ParamDir two
dbcopy -m modelOne -dbcopy.SetId 2 -p two
dbcopy -m redModel -s Default -p 101 -dbcopy.To db -dbcopy.ToDatabase "Database=dst.sqlite;OpenMode=ReadWrite"

Dbcopy create output directories (and json files) for model data by combining model name and run name or input set name. By default names combined with run id (set id) to make it unique. For example:

json file: modelName.run.1234.MyRun.json
directory: modelName/run.1234.MyRun

In case of output into csv by default directories and files combined with id's only if run name is not unique. To explicitly control usage of id's in directory and file names use IdOutputNames=true or IdOutputNames=false:

dbcopy -m modelOne -dbcopy.To csv
dbcopy -m modelOne -dbcopy.To csv -dbcopy.IdOutputNames=true
dbcopy -m modelOne -dbcopy.To csv -dbcopy.IdOutputNames=false

By default parameters and output results .csv files contain codes in dimension column(s), e.g.: Sex=[Male,Female]. If you want to create csv files with numeric id's Sex=[0,1] instead then use IdCsv=true option:

dbcopy -m modelOne -dbcopy.IdCsv
dbcopy -m modelOne -dbcopy.IdCsv -dbcopy.To csv
dbcopy -m redModel -dbcopy.IdCsv -s Default
dbcopy -m modelOne -dbcopy.IdCsv -dbcopy.RunId 101
dbcopy -m modelOne -dbcopy.IdCsv -dbcopy.TaskName taskOne

Dbcopy do auto detect input files encoding to convert source text into utf-8. On Windows you may want to expliciltly specify encoding name:

dbcopy -m modelOne -dbcopy.To db -dbcopy.CodePage windows-1252

If you want to write utf-8 BOM into output csv file then:

dbcopy -m modelOne -dbcopy.Utf8BomIntoCsv
dbcopy -m modelOne -dbcopy.Utf8BomIntoCsv -dbcopy.To csv

To delete from database entire model, model run results, set of input parameters or modeling task:

dbcopy -m modelOne -dbcopy.Delete
dbcopy -m modelOne -dbcopy.Delete -dbcopy.RunId 101
dbcopy -m modelOne -dbcopy.Delete -dbcopy.RunName MyFirstRun
dbcopy -m modelOne -dbcopy.Delete -dbcopy.SetId 2
dbcopy -m modelOne -dbcopy.Delete -s Default
dbcopy -m modelOne -dbcopy.Delete -dbcopy.TaskId 1
dbcopy -m modelOne -dbcopy.Delete -dbcopy.TaskName taskOne

OpenM++ using hash digest to compare models, input parameters and output values. By default float and double values converted into text with "%.15g" format. It is possible to specify other format for float values digest calculation:

dbcopy -m redModel -dbcopy.DoubleFormat "%.7G" -dbcopy.To db -dbcopy.ToDatabase "Database=dst.sqlite;OpenMode=ReadWrite"

By default dbcopy using SQLite database connection:

dbcopy -m modelOne

is equivalent of:

dbcopy -m modelOne -dbcopy.DatabaseDriver SQLite -dbcopy.Database "Database=modelOne.sqlite; Timeout=86400; OpenMode=ReadWrite;"

Output database connection settings by default are the same as input database, which may not be suitable because you don't want to overwrite input database.

To specify output database connection string and driver:

dbcopy -m modelOne -dbcopy.To db -dbcopy.ToDatabaseDriver SQLite -dbcopy.ToDatabase "Database=dst.sqlite; Timeout=86400; OpenMode=ReadWrite;"

or skip default database driver name "SQLite":

dbcopy -m modelOne -dbcopy.To db -dbcopy.ToDatabase "Database=dst.sqlite; Timeout=86400; OpenMode=ReadWrite;"

Other supported database drivers are "sqlite3" and "odbc":

dbcopy -m modelOne -dbcopy.To db -dbcopy.ToDatabaseDriver odbc -dbcopy.ToDatabase "DSN=bigSql"
dbcopy -m modelOne -dbcopy.To db -dbcopy.ToDatabaseDriver sqlite3 -dbcopy.ToDatabase "file:dst.sqlite?mode=rw"

ODBC dbcopy tested with MySQL (MariaDB), PostgreSQL, Microsoft SQL, Oracle and DB2.

Also dbcopy support OpenM++ standard log settings (described in wiki at http://www.openmpp.org/wiki/):

-OpenM.LogToConsole: if true then log to standard output, default: true
-OpenM.LogToFile:    if true then log to file
-OpenM.LogFilePath:  path to log file, default = current/dir/exeName.log
-OpenM.LogUseTs:     if true then use time-stamp in log file name
-OpenM.LogUsePid:    if true then use pid-stamp in log file name
-OpenM.LogNoMsgTime: if true then do not prefix log messages with date-time
-OpenM.LogSql:       if true then log sql statements into log file

Package Files

dbDelete.go dbToCsv.go dbToCsvExtra.go dbToCsvRun.go dbToCsvTask.go dbToCsvText.go dbToCsvWorkset.go dbToDb.go dbToDbRun.go dbToDbTask.go dbToDbWorkset.go dbToText.go dbToTextRun.go dbToTextTask.go dbToTextWorkset.go dbcopy.go textToDb.go textToDbRun.go textToDbTask.go textToDbWorkset.go

Package main imports 16 packages (graph). Updated 2020-02-03. Refresh now. Tools for package owners.