v28

Introducing MariaDB 10.1

The theme for v28 is to upgrade MariaDB 10.1.

We're very excited to bring this to cf-mysql, as it will unlock large improvements in stability and security. As always, we've encoded the best practices to upgrade from a MariaDB 10.0 install to MariaDB 10.1. You should be able to upgrade from a previous version of cf-mysql to v28 smoothly.

• cf-mysql now uses MariaDB 10.1.17 and Galera 25.3.17
  • [#128782093], [#130140531], [#128699855]

Server Configuration

We've continued to incorporate feedback from users to make additional server tweaks and defaults. These changes make the system better out of the box, and provide fewer surprises for those who aren't familiar with a Galera-based deployment.

• I'd like Galera clusters to be restricted to just the InnoDB storage engine [#121465037]
• Allow for override of the innodb_buffer_pool_instances [#111], [#123092551]
• As an Operator, I'd like to configure innodb_lock_wait_timeout so that I can tune my DB for OLTP or data-warehouse style workloads [#129591677]
• As an Operator, I'd like to configure character_set_server to something other than the 'utf8' default [#129592229]
• Bug: innodb_large_prefix disabled [#129589643]

Security

• Permissions for log files should not allow other users to read them [#122237641]
• Ensure logout destroys session cookies [#128616101]
• As an Operator, I want the super user account to be named root [#128692195]
  • Note: Based on user feedback, we'll make the administrator account name configurable again in a coming release. Sorry!

Manifest Generation

• Use new UAA style name/value manifest syntax for user creation [#113200331]

  • Now, you won't have to copy and paste your CF admin credentials into a cf-mysql stub.

• BOSH 2.0 Manifests

  If you've started to experiment with the new BOSH CLI, we've started to support BOSH 2.0-style manifests. In the future, we'll move to BOSH 2.0-style manifests entirely.

  • As an operator I would like to optionally leverage cloud-config when deploying cf-mysql-release to bosh-lite [#128799447]
  • As an operator I would like to optionally leverage links for IPs of mysql jobs when deploying cf-mysql-release to bosh-lite [#128799467]

Proxy Improvements

• As an Operator, I'd like the Proxies to more-deterministically choose which backend to route traffic [#128188319]
• Add a way to set `ulimit -n' for the proxy job [#112], [#123147571]
• Switchboard should log response from galera_healthcheck process when severing connections to a host [#128793743]

Download Logs Improvements

We've made some minor changes to the download logs script as well. This convenience script is an automated way to download logs from a variety of jobs without tedious bosh logs commands.

• As an Operator, I optionally want download-logs to additionally download and transmit audit and binlogs so that I can avoid transmitting sensitive data to unauthorized parties and save my sanity due to large scale artifacts. [#122885431]
• Bug: Unhelpful error calling download-logs without args, etc [#122176219]
• Bug: Unbound variable output_dir for download-logs script [#123227841]

Interestings

• As an Operator, I don't want to have the option to run the Acceptance Tests [#128466217]
  • Previous releases allowed the 'acceptance-tests' errand to run either the smoke tests or the full acceptance test suite. The latter is destructive, and there's no reason to run them outside of Continuous Integration (CI). Now, the errand is named smoke-tests and does the minimum necessary to validate a successful deployment.
• Bump CLI [#129406663]
  • Smoke tests now use a more recent version of the CLI, and a longer timeout to allow for slow DNS resolution.

Manifest Changes

Note: For those using the current bosh CLI, you'll need to set links in the new manifest to nil. Otherwise, you may see errors like:

Error 100: Unable to process links for deployment. Errors are:
...

In order to disable, anywhere the consumes: clause appears in your manifest, you must change the values to nil. Here's an example:

- name: mysql_z1
  templates:
  - name: mysql
    consumes:
      arbitrator: nil
      mysql: nil

Note: For those using the new bosh CLI, you'll need to use the guide to migrating instance groups to first-class AZs.

• Removed:
  • cf_mysql.mysql.admin_username, will be back in a future release
  • cf_mysql.acceptance_tests.smoke_tests_only, it is now always true
• Added:
  • cf_mysql.mysql.character_set_server, optional
  • cf_mysql.mysql.collation_server, optional
  • cf_mysql.mysql.innodb_buffer_pool_instances, optional
  • cf_mysql.mysql.innodb_lock_wait_timeout, optional
• Renamed:
  • cf_mysql.acceptance_tests.password renamed to cf_mysql.smoke_tests.password
  • cf_mysql.acceptance_tests.standalone_tests_only renamed to cf_mysql.smoke_tests.standalone_tests_only
  • cf_mysql.acceptance_tests.timeout_scale renamed to cf_mysql.smoke_tests.timeout_scale

v27

v27

The themes for this release are cluster stability, security and operability.

• cf-mysql v27 should include MariaDB 10.0.24 and Galera 25.3.14 [#114625673]

  Deprecation Warning: This will be the last cf-mysql-release to use MariaDB version 10.0. Subsequent releases will begin to use MariaDB 10.1, which became generally available late last year. Be aware that there may be compatibility differences between 10.0 and 10.1, so take care to test out changes before upgrading past cf-mysql-release v27.

• Note: cf-mysql now requires stemcells 3125 or more recent. As a security precaution, we recommend you update stemcells frequently.

• Note: If you've enabled syslogging, please see the security notice below regarding root credentials.

Pre-Start

Last year, we wrote a fussy story:

• As an Operator, I'd like configure an overall timeout which will break down into individual timeouts for each of the MySQL job's startup phases, so that large SST's can succeed [#100598304]

This condensed update.canary_watch_time, update.update_watch_time and DatabaseStartupTimeout into simply mysql_startup_timeout, and tried to intelligently split that up into sane defaults for each of the stages of database startup.

V27 moves much of this into the BOSH pre-start phase of the job lifecycle. This eliminates many timeout-related failures, although the system may still perform lengthy SSTs during monit start under certain circumstances.

• Leverage BOSH pre-start process [#100444318], [#114259781], [#117744313], [#117583221], [#117549509]

Defensive Enhancements

When restarting, a node re-syncs to the cluster using one of two methods, IST or SST. In the case of SST, it throws away its local data, and adopts the dataset of the cluster. We've found rare circumstances when that's not desirable. This release includes an optional Interruptor, which prevents a node from SST when doing so would delete its own database. We've also provided a rejoin-unsafe errand that automates the process of forcing a node to re-join the cluster after an Operator has determined that it's safe to do so.

• As an Operator, I do not want a possibly-sick node to re-join the cluster if it means there could be data loss. [#118416629], [#118416751], [#118793289]

Galera comes with a few known limitations. In an effort to try to protect users, and enhance stability, we've made the following changes.

• Ensure MyISAM experimental replication is disabled [#120629227]
• As an Operator, I don't want a Galera-based cluster to accept XA transactions [#121461877]
• As an Operator, I'd like my replication sums to be checked [#121008813]

The Operator, or automation, may to want to block all access to the MariaDB cluster, to enable backups, diagnosis, etc.

• As an Operator, I can enable and disable all traffic passing through the proxy [#127033173]

Security & Logging

More users are beginning to deploy cf-mysql into production. We've prioritized enabling logs and creating diagnostics that are useful to both Operators and Database Administrators to troubleshoot and audit an active cluster. We've also made changes to run more securely, in the event that an unauthorized user gains access to the virtual machines.

• Mariadb < 10.0.22 expose credentials in mysql.err.log when performing sst [#122147897]

  Note: cf-mysql-release v25 and earlier expose the credentials of the mysql root account. If you've enabled syslog, the root password will have been transmitted outside of the cf-mysql deployment. You must change the root password of your root account after upgrading to this version of cf-mysql-release.

• We've changed default syslog priority level from local1.error to the CAPI recommended standard of user.error.

• As an Operator, I'd like to be able to see the stdout/err of mariadb_ctl control script [#118612267]

• As a mysql operator, I would like to feel safe knowing that my mariadb process is not running as a privileged user (mysqld now runs as user vcap) [#115623381]

• As an operator, I do not want to see my mysql passwords in ps aux [#120031195]

• As an Operator, I'd like to be able to learn more about slow queries [#113421765]

• As an Operator, I'd like MySQL binlogs enabled on each Master node [#121089515]

• As an Operator, I'd like to enable additional debug logging to understand replication behavior. [#120996959]

• Log diagnostic cluster data to a CSV file [#118415807], [#121713351], [#127034351]

• As an Operator, I don't want users to be able to access databases whose names start with 'test' on a production deployment so I have confidence that my DB is secure by default [#96828790], [#117195649]

• Bug: cloudfoundry/cf-mysql-release #116: Enabling syslog doesn't appear to send mysql.err.log to syslog [#127031107]

• Bug: Log SST script output to mysql.err.log [#127664015]

Audit Logs

• cloudfoundry/cf-mysql-release #89: Enable audit_plugin [#113966825]
  • As an Operator, I'd like BOSH to preserve/rotate audit_logs [#118504463]
  • As an Operator, I do not want cluster_health_logger queries in the server audit log [#119423815]
  • Audit logs should be in persistent disk [#127034681], [#122210877]

Quota Enforcer

• As an Operator, I'd like the Quota Enforcer to connect as a non-root super user so that we can exclude the QE's activity from the audit logs [#118502911]
• quota-enforcer user should not need 'GRANT ALL PRIVILEGES' [#119029525]
• As an operator, I would like to provide a list of users that the quota enforcer will ignore [#119045503]
• As an Operator, I'd like the Quota Enforcer to run less frequently so that it doesn't fill up debugging log files. [#127033101]

Enhancements and Bug Fixes

• Update cf-mysql-release to Golang 1.7 [#128205943]
• As an operator, I want to configure the number of open files available to mysqld via max_open_files [#117110409]
• As an Operator, I'd like to have a property to change the name of the MySQL service in the Marketplace [#115805005]
• Enable innodb_large_prefix [#116578809]
• Length of time GRA files are kept around should be increased [#118606579]
• Bug: As an Operator, I want cf-mysql acceptance tests to work with recent versions of cf-release [#117671103]
• Bug: As an Operator, I'd like to change my database pre-seeds without running the bootstrap procedure [#122647541], [#127938125]
• Bug: Fix an out-of control CPU issue on the service broker VMs.
  • As an operator, I do not want monit to try and start mariadb_ctrl multiple times at the same time [#119498227]

Community Contributed Enhancements

• cloudfoundry/cf-mysql-release #87: Resolve package name, job name, and port conflicts [#113088085]
• cloudfoundry/cf-mysql-release #100: Add ability to override number of compilation workers [#117224143]
• Change the name of the deployment as it appears in bosh deployments: cloudfoundry/cf-mysql-release #101: Add ability to override deployment name [#117225385]
• cloudfoundry/cf-mysql-release #97: Trim release size [#116155371]
• cloudfoundry/cf-mysql-release #110: `./scripts/generate-deployment-m...

v26

• Don't miss Changes in Manifest Generation - Important!, below.

New Features

• CF MySQL now uses MariaDB 10.0.23 and Galera 25.3.9 [#110702918]

• As an Operator, I'd like to specify a user that has read-only-access to all data [#110369648]

  Supply a password for the roadmin user in the manifest, and cf-mysql will automatically provision a user with access to read all databases in the system, but permission to write to none. You can rotate this password by redefining it in the manifest and redeploying.

  • If you don't like this, it's easy to disable. [#112020565]

Introducing the Arbitrator

• Why build three when you can have two at only twice the price? [#2223556]

  For cf-mysql administrators who are careful with their infrastructure resources, the Arbitrator feature is a new deployment topology that uses a smaller VM footprint while maintaining high availability guarantees. Unlike the old three node topology, the Arbitrator decreases spend with no impact on performance.

  With cf-mysql v26, we've replaced one of the MySQL nodes with a lightweight Arbitrator node. Previously, the minimal HA configuration required three full-size MySQL nodes.

  When using only two nodes, the safest way to respond to a network or VM incident is downtime: the nodes stop accepting traffic until they can re-establish communication. An Arbitrator helps a two node MySQL cluster avoid the possibility of a split-brain condition or downtime by participating in weighted quorum elections.

  CF-MySQL scales best vertically, not horizontally. On many public clouds, large instances can be very expensive, but the Arbitrator uses a lightweight VM. By using an Arbitrator, as you scale your deployment, you spend only twice, not three times the cost required to support your applications. On AWS, we recommend a T2 small with no sacrifice in performance, see our benchmarking work below.

  For new deployments, the 2+1 topology is the default. You can find instructions to generate an install manifest in the README.md, and further instructions to migrate between Single, Three, and 2+1 toplogies in the Arbitrator documentation. The instructions also cover how to migrate between each. If you follow along carefully, you won't experience any downtime.

  • [#109744228], [#109744232], [#109744230], [#109744234, [#109744236], [#110772724], [#111741306], [#109748868]
    • You can view our results in the Benchmarking spreadsheet.

Bugfixes and Interestings

• Enable wsrep_load_data_splitting option [#112648647]
  • Ensures bulk-loading data from big files using LOAD DATA INFILE uses transactions.
• Change default of skip_name_resolve to true [#106676160]
  • As promised in the v25 release notes.
• Bugfix: Bump route-registrar libraries [#112556657]
• Bugfix: As an Operator and as a Service Author, it'd be great to keep a local copy of logs when syslog is configured [#111344064]
• Bugfix: wsrep_node_name is the same for all mysql jobs [#113135203]
• Docs fix: Document that user should wait if they encounter the monit 503 error during bootstrap [#111778592]

Changes in Manifest Generation - Important!

• As an Operator, I should be able to specify flags instead of positional args when generating a manifest [#110775864].

  We've changed the way you create manifests considerably. The new way closely resembles how diego generates manifests. Regrettably, for those of you who do not generate manifests by updating stubs and re-generating manifests, the new manifests are significantly different than versions pre-v26.

  The generate-deployment-manifest script is now located in the cf-mysql-release scripts directory. Run ./scripts/generate-deployment-manifest without argument to see help output.

  • Manifest generation now requires spiff v1.0.7

  If you're the type of person who manually updates deployment manifests, we've unfortunately made a lot of work for you. We highly recommend you switch to generating manifests from the provided stubs each time you update cf-mysql-release.

Manifest Changes

MySQL Job Spec Changes

• Move skip_name_resolve to cf_mysql.mysql.skip_name_resolve and it now defaults to true
• cf_mysql.mysql.roadmin_password combined with cf_mysql.mysql.roadmin_enabled determine if the read only user exists
  • If roadmin_password is blank or roadmin_enabled is false, the read only user will be disabled
• port is now cf_mysql.mysql.port
• healthcheck_port is now cf_mysql.mysql.healthcheck_port
• max_connections is now cf_mysql.mysql.max_connections
• innodb_buffer_pool_size is now cf_mysql.mysql.innodb_buffer_pool_size
• cluster_ips is now cf_mysql.mysql.cluster_ips
• max_heap_table_size is now cf_mysql.mysql.max_heap_table_size
• tmp_table_size is now cf_mysql.mysql.tmp_table_size
• wsrep_max_ws_rows is now cf_mysql.mysql.wsrep_max_ws_rows
• wsrep_max_ws_size is now cf_mysql.mysql.wsrep_max_ws_size
• gcache_size is now cf_mysql.mysql.gcache_size
• ib_log_file_size is now cf_mysql.mysql.ib_log_file_size
• seeded_databases is now cf_mysql.mysql.seeded_databases
• bootstrap_endpoint.username is now cf_mysql.mysql.bootstrap_endpoint.username
• bootstrap_endpoint.password is now cf_mysql.mysql.bootstrap_endpoint.password

Proxy Job Spec Changes

• Move all properties to be under the top level cf_mysql key, except for network_name
• cluster_ips now refers to the same cf_mysql.mysql.cluster_ips from the mysql job spec
• Rename the entire proxy hash to cf_mysql.proxy
• added cf_mysql.proxy.arbitrator_ip property to tell the proxy where the arbitrator lives. This, in case you are deploying using an arbitrator, see above for details.

CF MySQL Broker Job Spec Changes

• api_url is now cf.api_url

• skip_ssl_validation is now cf.skip_ssl_validation

• mysql_node.host is now cf_mysql.external_host

• Move auth_username, auth_password, cookie_secret, max_user_connections_default, services, ssl_enabled to

  cf_mysql:
    broker:
      auth_username:
      auth_password:
      cookie_secret:
      max_user_connections_default:
      services:
      ssl_enabled:
  

• Rename mysql_node hash to cf_mysql.mysql

Arbitrator Job Spec

• See spec file spec file for manifest properties.
• These properties are optional dependent upon whether you are deploying an arbitrator.

Acceptance Tests Errand Spec Changes

• cf.apps_domain is now cf.app_domains
• Add cf.smoke_tests.org
• Add cf.smoke_tests.use_existing_org
• smoke_tests_only is now cf_mysql.acceptance_tests.smoke_tests_only
• standalone_tests_only is now cf_mysql.acceptance_tests.standalone_tests_only
• timeout_scale is now cf_mysql.acceptance_tests.timeout_scale
• Move all proxy properties under cf_mysql.proxy
• Add cf_mysql.broker.services

Bootstrap Errand Spec Changes

• All properties have been moved under cf_mysql.mysql

Broker Registrar Errand Spec Changes

• broker.host is now cf_mysql.external_host
• Remove broker.name
• broker.protocol is now cf_mysql.broker.brotocol
• broker.services is now cf_mysql.broker.services
• broker.username is now cf_mysql.broker.auth_username
• broker.password is now cf_mysql.broker.auth_password
• broker.protocol is now cf_mysql.broker.protocol
• broker.port is now cf_mysql.broker.port

Broker Deregistrar Errand Spec Changes

• broker.name is removed
• Added cf_mysql.broker.services

v25.1

Bugfix

• Quota-enforcer should use GRANT statements to ensure replication [#111536932]

  In collaboration with a keen-eyed user, we determined that the Quota Enforcer was incorrectly restoring write privileges. As a result, these changes were not replicated properly across the cluster.

No new features or other changes have been introduced in this release.

Special Note

• We noticed this issue when the Quota Enforcer began to make changes to an account created by the user directly, not via the cf CLI. To be clear, cf-mysql-release does not support user-created account. Please use either cf bind-service or cf create-service-key to create accounts.

  We will add an optional global read-only user to a feature release of cf-mysql in the future. [#110369648]

v25

New Features

• As an Operator, I'd like to trigger the bootstrap process without performing the work manually. [#106994020]
  • This mini-epic automates the process of bootstrapping the MariaDB/Galera cluster via a bosh errand. Bootstrapping is only necessary when the MariaDB VMs have been interrupted, either by reboot or network interruption. See the documentation for more information.
• Updated MariaDB to version 10.0.22 [#106993116]
• As an Operator, I'd like to optimize connection performance by skipping reverse DNS resolution [#106063730]
  • We've introduced a new configurable property, skip_name_resolve for the mysql jobs. The current default is 'false,' the same behavior of previous versions of cf-mysql-release. When set to 'true,' this option improves performance by skipping DNS resolution to enforce hostname restrictions. In the typical Cloud Foundry context, hostname restrictions are not used.
    • Note: The next release of cf-mysql-release will change the default to 'true'.
• As an Operator, I'd like to be able to declare service plans as private in the deployment manifest [#103699052]
  • Add support for private plans. In previous versions, all plans were automatically published to all CF organizations by the broker-registrar errand. This feature introduces a new optional property, private. When set to 'true,' plans will not be published to all CF orgs when running broker-registrar. Private plans must be manually published by a cf admin, as described in the documentation.

New Documentation

• As a BOSH Operator, I should discover that deployment fails if plan names do not conform to CLI format [#107603894]
• As an Application Developer, I'd like to see a link to the documentation when I run cf service [instance name] [#107963372]
• Replace manual bootstrap instructions in OSS docs with run errand instructions [#109827548]

Bugfixes and Interestings

• Quota enforcer will spawn a new process when restarted without killing the existing process [#99640614]
  • Addresses a bug in which the Service Broker VMs may be overloaded by hordes of Quota Enforcer processes.
• A wildcard SSL cert does not cover proxy-0.p-mysql.domain.com [#95581184]
  • We've changed the proxy dashboard URL from "proxy-0.p-mysql.DOMAIN" to "proxy-0-p-mysql.DOMAIN." This subtle change makes the endpoint easier to cover with an SSL wildcard certificate.
• Service broker should update max user connections on startup [#109492744]
  • Fix bug where max_user_connections quota for existing service instances was not updated when a plan definition is updated.
• Service broker does not update existing service instances when the size of a plan is updated [#105336102]
  • Fix bug where storage quota for existing service instances was not updated when a plan definition has been changed.
• broker-registrar errand should update broker if it already exists [#95587238]
  • The cf update-service-broker command is now run automatically as part of broker-registrar errand. This saves the operator from having to run CF commands manually after adding/modifying plans.
• Switchboard should not start if proxy or api section fail to run [#87746750]
• Update script should fail fast if permissions denied [#105491860]
• Move concourse config into cf-mysql-ci repo [#105428042]
  • We've separated out our Continuous Integration assets into a separate repository for easy access. Try Concourse today!

Manifest Changes

• Manifest changes required for bootstrap security [#106993928]
  • Added new properties bootstrap_endpoint.username and bootstrap_endpoint.password.
  • To enable the auto-bootstrap feature, you'll find a new section in the Infrastructure stubs provided with the release. You'll need to provide a new username and password for the bootstrap endpoints.

  bootstrap_endpoint:
      username: REPLACE_WITH_A_USERNAME_YOU_MAKE_UP
      password: REPLACE_WITH_A_PASSWORD_YOU_MAKE_UP

• Manifest changes required to declare plans private [#103699052]
  • Service plans are typically defined in plans_stub.yml. We've added a new property, private. To configure, please read the documentation on service plans.
• After updating your stubs, make sure to generate a new manifest as described in Create Manifest and Deploy.

v24

Bug Fixes

• Fixed bug in v23 whereby service dashboard failed when Cloud Controller is configured with a self-signed SSL certificate; e.g. when deploying to BOSH Lite. [#105326528]

v23

Updated MariaDB

• Operator should be able to verify that cf-mysql-release includes MariaDB 10.0.21 and Galera library 25.3.9 [#99853996]

Bug Fixes

• CVE-2015-3900: MITM rubygems vulnerability [#101620234]
• An Operator should be able to deploy cf-mysql on vSphere using the provided sample stub [#100774212]
• cloudfoundry/cf-mysql-release #80: bosh deploy error : no implicit conversion of Fixnum into String [#102881976]

Switchboard:

• Switchboard fails to find recreated mysql node when ARP cache locked by hanging SYN_SENT [#101784830]
• Every instance of Switchboard registrars the route proxy-0.p-mysql. rather than changing based on AZ index [#101778816]
• Regression: switchboard returns 500 error when fetching list of backends [#103439908]

Startup:

We've done some work to ensure smooth startup for databases that have very large data sets. To understand these changes, please read the bosh documentation covering the canary_watch_time and update_watch_time properties.

• As an Operator, I want to configure how long the startup script waits for the database to come online [#100442046]
• During long SSTs, BOSH reports job as failing forever, mysqld is not properly managed by its control script. [#100600470]
• As an Operator, I'd like configure an overall timeout which will break down into individual timeouts for each of the MySQL job's startup phases, so that large SST's can succeed [#100598304]

Compiling and Testing

• cf-mysql acceptance tests should use https only [#101143514]

• cf-mysql-broker compilation fails if workstation has mysql installed rather than mariadb [#101706476]

• mysql-broker fails because of mysql2 gem packaging issue [#98972188]

• Native extensions for mysql2 gem should be build during compilation stage [#101784138]

• cloudfoundry/cf-mysql-release #79: allow configureable smoke test password [#102174066]

• Users that have built older dev releases of cf-mysql prior to updating to v23 may encounter the following error when running ./update:

  error: The following untracked working tree files would be overwritten by checkout:
    packages/ruby/packaging
    packages/ruby/spec
  Please move or remove them before you can switch branches.
  

  This can be fixed by running rm -r ./packages/ruby && ./update.

Documentation Updates

• New document: Known Issues
• Clarify that additional cf security groups are not required to run cf-mysql since cf v211 [#100047222]
• As an Operator, I'd like the sample stubs to follow the convention of using REPLACE for every place a stub should be modified [#99473360]
• Add [#100600470] as a known issue in cf-mysql-release [#100598530]
• All templates should have the same comment/description about meta:environment property [#99474804]

Standalone

We've done some work to allow cf-mysql-release to start up without relying on a running cf-release. This work is still in progress, but here are the stories that have been completed: [#84945758], [#84945466]], and [#99638360]

Manifest Changes

Note: Due to changes in our manifests, cf-mysql-release v23 requires spiff v1.0.7 or higher.

• MySQL job
  • Added property database_startup_timeout
    
    Defines how long the startup scripts wiat for the database to come online (in seconds)
• Proxy job
  • Added property proxy.proxy_ips
  • Added property standalone
• Acceptance-tests job
  • Added property proxy.proxy_count (defaults to 2)
  • Added property smoke_test_password and gave it a somewhat complex default
    
    In response to GH Issue #79, which validates a password minimum complexity feature now available in Cloud Foundry.

Special Note for bosh-lite Only

If you encounter an error when deploying:

Error 400007: `cf-mysql-broker_z1/0' is not running after update

Check to see that your error output matches the example below. If they match, then you'll need to update your cf-release to v219 or more recent. In order to improve stability, cf-release switched the system domain from 10.244.0.34.xip.io to bosh-lite.com. In #104772322, we've updated the manifest system domain to point to bosh-lite.com as well. The settings in both cf-release and cf-mysql-release must match.

Jobs
proxy_z1
  properties
    ± external_host:
      - p-mysql.10.244.0.34.xip.io
      + p-mysql.bosh-lite.com
proxy_z2
  properties
    ± external_host:
      - p-mysql.10.244.0.34.xip.io
      + p-mysql.bosh-lite.com
cf-mysql-broker_z1
  properties
    ± cc_api_uri:
      - https://api.10.244.0.34.xip.io
      + https://api.bosh-lite.com
    ± external_host:
      - p-mysql.10.244.0.34.xip.io
      + p-mysql.bosh-lite.com
cf-mysql-broker_z2
  properties
    ± cc_api_uri:
      - https://api.10.244.0.34.xip.io
      + https://api.bosh-lite.com
    ± external_host:
      - p-mysql.10.244.0.34.xip.io
      + p-mysql.bosh-lite.com
broker-registrar
  properties
    broker
      ± host:
        - p-mysql.10.244.0.34.xip.io
        + p-mysql.bosh-lite.com
    cf
      ± api_url:
        - https://api.10.244.0.34.xip.io
        + https://api.bosh-lite.com
      ± apps_domain:
        - 10.244.0.34.xip.io
        + bosh-lite.com
broker-deregistrar
  properties
    cf
      ± api_url:
        - https://api.10.244.0.34.xip.io
        + https://api.bosh-lite.com
      ± apps_domain:
        - 10.244.0.34.xip.io
        + bosh-lite.com
acceptance-tests
  properties
    broker
      ± host:
        - p-mysql.10.244.0.34.xip.io
        + p-mysql.bosh-lite.com
    cf
      ± api_url:
        - https://api.10.244.0.34.xip.io
        + https://api.bosh-lite.com
      ± apps_domain:
        - 10.244.0.34.xip.io
        + bosh-lite.com
    proxy
      ± external_host:
        - p-mysql.10.244.0.34.xip.io
        + p-mysql.bosh-lite.com

Properties
± app_domains:
  - 10.244.0.34.xip.io
  + bosh-lite.com
cf
  ± api_url:
    - https://api.10.244.0.34.xip.io
    + https://api.bosh-lite.com
  ± apps_domain:
    - 10.244.0.34.xip.io
    + bosh-lite.com
± domain:
  - 10.244.0.34.xip.io
  + bosh-lite.com

Meta
No changes


Deploying
---------

Director task 293
  Started preparing deployment
  Started preparing deployment > Binding deployment. Done (00:00:00)
  Started preparing deployment > Binding releases. Done (00:00:00)
  Started preparing deployment > Binding existing deployment. Done (00:00:00)
  Started preparing deployment > Binding resource pools. Done (00:00:00)
  Started preparing deployment > Binding stemcells. Done (00:00:00)
  Started preparing deployment > Binding templates. Done (00:00:00)
  Started preparing deployment > Binding properties. Done (00:00:00)
  Started preparing deployment > Binding unallocated VMs. Done (00:00:00)
  Started preparing deployment > Binding instance networks. Done (00:00:00)
     Done preparing deployment (00:00:00)

  Started preparing package compilation > Finding packages to compile. Done (00:00:00)

  Started preparing dns > Binding DNS. Done (00:00:00)

  Started preparing configuration > Binding configuration. Done (00:00:01)

  Started updating job proxy_z1 > proxy_z1/0 (canary). Done (00:00:41)
  Started updating job proxy_z2 > proxy_z2/0 (canary). Done (00:00:42)
  Started updating job cf-mysql-broker_z1 > cf-mysql-broker_z1/0 (canary). Failed: `cf-mysql-broker_z1/0' is not running after update (00:05:14)

Error 400007: `cf-mysql-broker_z1/0' is not running after update

v22

New Features and Configuration Changes

• cf-mysql-release now includes MariaDB 10.0.19 #94191012
• Specify MySQL plans in a spiff stub so that they can be customized without editing/saving/restoring my entire Bosh manifest #95585028
• Configuration to avoid simultaneous large queries from failing due to a full /tmp #94691776
• Expose Galera configuration options in the manifest so that large database transactions can complete without error #98155338
• Galera healthcheck now returns all Galera states so that the Operator can better reason about the state of the system #93641798
• Enable a configuration to enable the cluster to recover from power failure #90573188
• Security: Require that traffic to the switchboard dashboard be encrypted #95181616
• Security: Run MySQL deployment as user vcap, not root #95011868

New Documentation

• Scaling options of the MySQL service components #95275330
• How the service broker and quota enforcer treat oversubscription #94248700
• Explain how configuring zero proxies with deployments of cf-mysql-release means that all apps will need to be re-bound in the case of a state change #92844780

Bugfixes and Interestings

• v20 missing documentation: services1 network #97153152
  
  Note: We've renamed the services1 network to compilation
• MySQL broker dashboard should not return 500 if OAuth access token expires #96009586

Special Note for Amazon AWS users

We weren't satisfied with the difficulty required to configure cf-mysql-release across multiple AWS AZ's, so we've put some effort into making the stub much easier to follow. For v22, please feel free to use this pre-release version of the stub that will appear in v23. It's been tested and validated to work with v22: http://bit.ly/v23-sample-aws-stub

• The AWS sample stub follows the convention of using REPLACE for every place a stub should be modified #99473360
  

New Documentation for Community Developers

• Project CONTRIBUTING.md now explains develop / master / release-candidate #95250996

v21 - DO NOT USE

Final release version 21

v20

Highlights

• Support AWS multi-availability zone deployments #87241160
  
  See below for upgrade instructions.
  • Added networks and resource pools for Multiple AZs
  • Updated job names for Multiple AZs
• Operator can now configure cf-mysql-release with different instance types for each job #87242348
• Improved Operator-visible logging from mariadb_ctrl #88102590
• We have updated MariaDB to version 10.0.17, Galera to 25.3.9, and Xtrabackup to 2.2.10 #90127848, #94023216
• use xtrabackup-v2 as replication mechanism (improved stability during replication) #88835488
• Compile against the MariaDB connector instead of additionally including the MySQL connector #92725528
• We have rewritten the Quota Enforcer to improve stability #88665590

New Documentation

• How to upgrade a MySQL service plan #91748318
• Specify instance sizing in BOSH manifests: Deployment Resources #89325154

Bug Fixes and Miscellaneous Changes

• Once over quota, write privs are not restored by dropping all tables #86454200
• Upgrade sample stub to allow self signed certificates #82316314
• CF-MySQL service broker must speak to the CC API via SSL #93485852
• Update Ruby version for service broker - address any CVEs #93485852
• Upgrade Rails and other gems to latest version for broker repos #86044548
• Configurable defaults for MariaDB in-memory behavior #94043690
  

  max_heap_table_size = 16777216 # Rows
  tmp_table_size = 33554432 # Bytes

• Compilation instances now use new instance types on AWS #89434250
• Added small bosh-lite helper scripts
• Improved MySQL node startup speed
  • Nodes come online more quickly
  • Nodes fail faster if sub-processes fail
• Enabled option to run acceptance-tests errand against pre-existing org (instead of creating a random one each time) #90070764

v20 Upgrade Instructions for AWS #93040612

If you've deployed v19 or earlier, there are specific manifest changes you'll need to make to take advantage of the new multi-AZ templates. You'll need to rename the mysql job to mysql_z1; this is a one-time change as you upgrade to v20. Future versions of cf-mysql-release will continue to use the _z1, _z2, _z3 pattern.

1. Download existing manifest
   

   $ bosh download manifest [your deployment name] cf-mysql-deployment.yml

2. Set manifest to newly-downloaded manifest
   

   bosh deployment cf-mysql-deployment.yml

3. Edit the manifest to rename mysql job to mysql_z1
   

   $ bosh edit deployment

4. Rename job
   

   $ bosh rename job mysql mysql_z1

5. Create a stub file called cf-mysql-aws-stub.yml by copying and modifying the sample_aws_stub.yml in templates/sample_stubs.
   

   $ cp templates/sample_stubs/sample_aws_stub.yml cf-mysql-aws-stub.yml

6. Make a new manifest with multiple availability zones
   

   $ ./generate_deployment_manifest aws cf-mysql-aws-stub.yml > cf-mysql-new.yml

7. Set manifest to newly-created manifest
   

   $ bosh deployment cf-mysql-new.yml

8. Edit manifest
   

   $ bosh edit deployment

   Configure only one node of mysql_z1 and 0 of mysql_z2 and mysql_z3
   • Set instances of mysql_z2 and mysql_z3 to 0
   • Remove static ips from mysql_z2 and mysql_z3
   • Set cluster IPs on mysql_z1 to just the IP of the mysql_z1 instance
   • Set cluster IPs on proxy jobs to just the IP of the mysql_z1 instance
9. Deploy (resulting in a single mysql node, 2 brokers and 2 proxies correctly distributed across AZs)
   

   $ bosh deploy

10. Validate by running bosh vms cf-mysql. Your output should look similar to this:
    

    +----------------------+---------+--------------------+-------------+
    | Job/index            | State   | Resource Pool      | IPs         |
    +----------------------+---------+--------------------+-------------+
    | cf-mysql-broker_z1/0 | running | cf-mysql-broker_z1 | 10.10.3.101 |
    | cf-mysql-broker_z2/0 | running | cf-mysql-broker_z2 | 10.10.4.101 |
    | mysql_z1/0           | running | mysql_z1           | 10.10.3.10  |
    | proxy_z1/0           | running | proxy_z1           | 10.10.3.11  |
    | proxy_z2/0           | running | proxy_z2           | 10.10.4.11  |
    +----------------------+---------+--------------------+-------------+
    

11. Make a new manifest using templates, as above (leave unchanged at 3 mysql nodes, etc).
12. Set manifest to newly-created manifest, as above.
13. Deploy (resulting in 3 mysql nodes, 2 proxies and 2 brokers distributed across multiple AZs), as above.
    Bosh should show you manifest changes that look similar to this:
    

Jobs
mysql_z1
  properties
    ± cluster_ips: 
      + 10.10.4.10
      + 10.10.5.10
mysql_z2
  ± instances: 
    - 0
    + 1
  ± networks: 
    - {"name"=>"mysql2", "static_ips"=>nil}
    + {"name"=>"mysql2", "static_ips"=>["10.10.4.10"]}
mysql_z3
  ± instances: 
    - 0
    + 1
  ± networks: 
    - {"name"=>"mysql3", "static_ips"=>nil}
    + {"name"=>"mysql3", "static_ips"=>["10.10.5.10"]}
proxy_z1
  properties
    ± cluster_ips: 
      + 10.10.4.10
      + 10.10.5.10
proxy_z2
  properties
    ± cluster_ips: 
      + 10.10.4.10
      + 10.10.5.10

1. Validate by running bosh vms cf-mysql. Your output should look similar to this:

   +----------------------+---------+--------------------+-------------+
   | Job/index            | State   | Resource Pool      | IPs         |
   +----------------------+---------+--------------------+-------------+
   | cf-mysql-broker_z1/0 | running | cf-mysql-broker_z1 | 10.10.3.101 |
   | cf-mysql-broker_z2/0 | running | cf-mysql-broker_z2 | 10.10.4.101 |
   | mysql_z1/0           | running | mysql_z1           | 10.10.3.10  |
   | mysql_z2/0           | running | mysql_z2           | 10.10.4.10  |
   | mysql_z3/0           | running | mysql_z3           | 10.10.5.10  |
   | proxy_z1/0           | running | proxy_z1           | 10.10.3.11  |
   | proxy_z2/0           | running | proxy_z2           | 10.10.4.11  |
   +----------------------+---------+--------------------+-------------+