Release Notes 0.4.0

Hydra 0.4.0

Major changes

• Hydra stages will now restart cleanly on Windows.
• Configuration has changed. Major differences are:
  • Specify the MongoDB to use with database.url, of format mongodb://host:port,host2:port2
  • The pipeline/namespace property has changed named to database.pipeline
  • Connectors and others that use the database library can now use an existing MongoConfiguration instead of having to create their own implementation
• Hydra can now connect to MongoDB replica sets.
• Stages can now have a timeout set on their processing. A stage that times out will destroy its JVM, forcing a restart of all stages in that group. The document will be failed.
• The LocalDocument class used in stages now has some utility methods to reduce boiler-plate code concerning getting objects of the correct type. There is also an appender for list fields.
• You can now fetch all attachments for a document using RemotePipeline.getFiles() in a stage.
• The Admin Service now takes a configuration file similar to the Core. Set your configuration directory with the system variable hydra.admin.config.dir, and put your admin-service.properties there.
• The Admin Service now accepts JSONP requests on all endpoints.
• Dates in Hydra are now serialized with milliseconds intact.
• The Admin Service will now respond with proper HTTP errors and JSON body, instead of dropping Internal Server Error.
• The CmdLineInserter has been moved to a separate project called tools. The jar has been renamed to hydra-inserter.jar.

Upgrading from 0.3.x

• Make sure your configuration file has been updated - see resource.properties in the release
• RegexStage and SetStaticFieldStage have changed behaviour to be more general. See the class comments and stage descriptions for more details.
• It is recommended to recompile your stages with the 0.4.0 API, but it should not be necessary (as long as you are not mixing API versions in the same stage group).

Resolved issues

https://github.com/Findwise/Hydra/issues?milestone=3&page=1&state=closed

Pull requests merged into 0.4.0

• #286 remen/279_process_timeout A timeout in a processor now unconditionally restarts the system process
• #283 laserval/admin-service-stage-config-attachment-fix Adds check for empty configuration to Admin Service stage config endpoint
• #282 andreassalomonsson/continueregex Continue instead of return in RegexStage
• #280 laserval/inserter-project Separate Inserter project
• #277 laserval/init-exception Adds more generic exception to stages' init() method
• #276 laserval/remotepipeline-getfiles Adds method for retrieving all document attachments, changes Date serialization
• #275 laserval/localdocument-utility-methods Adds utility methods for getting and appending to content fields
• #274 laserval/solr-out-latest-solrj Updates solrj dependency to latest release
• #270 staunsbjerg/staunsbjerg-concurrent-memorycache-fix Fixed concurency issue #268 in MemoryCache
• #264 laserval/ssimon-process-threads AbstractProcessStage timeouts and cleaner shutdown
• #262 laserval/fix-259-java6-compilation-error Fix for #259: Removes Java 7-specific constructor
• #261 Nycander/fix-junit-dep Brings test dependencies up to date
• #260 ebbesson/jsonp-support Adds Jsonp support to Admin Service. Jetty:run build option to admin service as well as resolves issue #259
• #258 laserval/mongodb-client-update MongoDB client update and database configuration fixes
• #257 laserval/fix-abstractstage-parameter-checks Fixes to stage parameter parsing
• #253 Nycander/find-unknownhost Fixes failing test when you are on a network which recognizes unknownhost as a valid hostname
• #252 Nycander/set-static-fields-strict-empty-check SetStaticFieldStage no longer checks for the mere existance of a field, but looks at the value to see if it is empty as well
• #243 laserval/fix-windows-stage-restarts Fixes #227, stage restarts on Windows should no longer cause repeated failures
• #242 laserval/fix-tika-stage-wiremock-port-collision Changes port for SimpleFetchingTikaStage wiremock to be a less common one
• #237 remen/#235_mockhttpintika Use wiremock in SimpleFetchingTikaStageTest
• #233 Nycander/tika-fetch-close-resources SimpleFetchingTikaStage should close its resources after fetching
• #231 laserval/stage-update-fix Fixes one cause of #168
• #228 ssimon/master Getting started simplified
• #225 laserval/admin-service-configuration Admin-service configuration via property file
• #224 rgraneru/admin-service-delete-documents Admin service delete documents
• #221 jwestberg/solr-required Removes required from SolrOutputStage parameters. Fixes #220
• #218 laserval/mongodb-junit-external-resources jUnit external resources
• #216 jwestberg/stage-group-classpath Fixes bug where -jar and -cp were conflicting and breaking stagegroups

Contributors

• @andreassalomonsson
• @ebbesson
• @jwestberg
• @laserval
• @Nycander
• @remen
• @rgraneru
• @ssimon
• @staunsbjerg